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ABSTRACT 


The  objective  of  this  review  is  to  better  understand  the  current  state-of-the-art  methods 
for  representing  uncertainty  in  sensor-fusion  data  using  graphical,  visual  displays  to 
inform  military  decision  making.  Decision  makers  in  Maritime  defence  and  security  often 
consider  many  sources  of  information  when  trying  to  develop  an  understanding  of  the 
situation  within  their  area  of  responsibility.  Information  from  each  source  has  some 
uncertainty,  as  does  their  integrated  or  “fused”  summary  result.  The  purpose  of 
presenting  uncertainty  is  to  enhance  understanding  of  where  the  contact  is,  has  been 
and  will  be  so  that  the  operator  is  able  to  make  better  decisions  about  the  contact’s 
intent.  The  scientific  literature  presents  a  number  of  approaches  to  graphical  information 
visualization  that  improve  comprehension  of  uncertainty  using  intrinsic  or  extrinsic 
modification  of  graphical  elements.  However,  little  of  the  existing  research  shows  this 
understanding  results  in  improved  decision  making  in  the  Maritime  domain.  Few 
commercial  vendors  of  display  systems  provide  uncertainty  estimates  in  their  tactical 
plots  and  it  is  not  clear  whether  this  feature  would  be  useful  to  tactical  Maritime 
operators.  This  report  suggests  a  Human  Factors  analysis  approach  to  clarify  where 
uncertainty  visualization  may  be  useful  in  Maritime  defence  and  security,  followed  by 
recommendations  of  extrinsic  and  intrinsic  uncertainty  visualization  techniques  that 
could  be  used  to  validate  the  analysis  experimentally. 
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RESUME 

Le  present  examen  a  pour  but  de  mieux  comprendre  les  methodes  de  pointe  utilisee  en 
ce  moment  pour  representer  I’incertitude  associee  a  la  fusion  des  donnees  de  capteurs 
a  I’aide  d’affichages  graphiques  et  visuels  afin  d’eclairer  la  prise  de  decision  militaire. 
Les  decideurs  du  domaine  de  la  defense  et  de  la  securite  maritimes  doivent  souvent 
prendre  en  compte  des  renseignements  d’une  diversity  de  sources  lorsqu’ils  cherchent 
a  mieux  comprendre  une  situation  dans  leur  zone  de  responsabilite.  Les 
renseignements  de  chaque  source  comprennent  tous  un  certain  niveau  d’incertitude,  et 
il  en  est  egalement  ainsi  pour  les  resultats  sommaires  integres  ou  «  fusionnes  ». 
Presenter  I’incertitude  permet  d’ameliorer  la  comprehension  de  I’endroit  ou  se  trouve  le 
contact,  ou  il  a  ete  et  ou  il  se  dirige  afin  que  I’operateur  soit  en  mesure  de  prendre  de 
meilleures  decisions  au  sujet  de  I’intention  du  contact.  La  litterature  scientifique 
presente  certaines  methodes  de  representation  d’information  graphique  qui  ameliorent 
la  comprehension  de  I’incertitude  en  apportant  des  modifications  intrinseques  ou 
extrinseques  aux  elements  graphiques.  Cependant,  peu  de  recherches  demontrent  que 
cette  comprehension  des  resultats  entrame  une  amelioration  de  la  prise  de  decisions 
dans  le  domaine  maritime.  Un  nombre  restreint  de  fournisseurs  commerciaux  de 
systemes  d’affichage  offrent  des  estimations  de  I’incertitude  dans  leurs  traces  tactiques; 
or,  il  n’est  pas  certain  que  cette  fonction  serait  utile  aux  operateurs  maritimes  tactiques. 
Le  present  rapport  suggere  d’entreprendre  une  analyse  des  facteurs  humains  afin  de 
determiner  clairement  ou  la  representation  visuelle  de  I’incertitude  est  utile  dans  le 
domaine  de  la  defense  et  la  securite  maritimes,  et  il  presente  des  recommandations  de 
techniques  de  visualisation  extrinseques  et  intrinseques  qui  pourraient  servir  a  valider 
I’analyse  de  maniere  experimentale. 
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1  INTRODUCTION 

This  document  is  one  of  two  contractual  deliverables  that  summarize  the  work  done  on  W7707- 
145734/001 /HAL  “Visualizing  uncertainty  for  data  fusion  graphics”,  conducted  for  Defence 
Research  and  Development  Canada  (DRDC.)  The  work  involves  a  review  of  selected  scientific 
literature  and  a  discussion  of  some  current  industry  solutions  for  visualizing  the  uncertainty  of 
sensor  data  fusion,  particularly  in  the  Maritime  defence  and  security  domain.  The  second 
deliverable,  a  Microsoft  PowerPoint  presentation  will  be  provided  separately. 

A  bibliographic  database  of  related  literature  that  was  created  in  Mendeley1  and  Adobe  Portable 
Document  Format  (PDF)  copies  of  the  reviewed  documents  will  also  be  provided  to  the  client, 
although  they  are  not  contractual  deliverables. 

1.1  Background 

Decision  makers  within  the  Canadian  Armed  Forces  (CAF)  and  the  Department  of  National 
Defence  (DND)  often  consider  many  sources  of  information  when  trying  to  develop  an 
understanding  of  the  situation  within  their  area  of  responsibility  (AOR).  It  is  generally  accepted 
(although  perhaps  not  proven)  that  decision  makers  should  be  aware  of  the  degree  of 
uncertainty  associated  with  each  relevant  piece  of  important  information  and  consider  the 
uncertainty  when  forming  hypotheses  or  judgements  their  understanding  current  operational 
context. 

Martineau  and  Roy  (2011)  describe  the  Maritime  defence  and  security  domain  where  decision 
makers  collect  information  to  infer  the  intent  of  contacts  operating  in  their  AOR,  both  civilian  and 
military.  Most  of  these  contacts  have  legitimate  reasons  for  operating  in  the  AOR,  but  some  may 
be  acting  for  illegal  or  unsanctioned  reasons.  Maritime  defence  and  security  operators  attempt 
to  detect  and  deal  with  these  undesirable  contacts,  identifying  them  from  a  sea  of  distractor, 
legitimate  contacts.  Data  and  information  used  to  detect  or  identify  contacts  may  come  from  any 
of  a  number  of  sources  including  ship  registries,  voluntary  reporting  or  remote  sensing  in  their 
situation  assessment  procedures  (Lapinski,  2010).  Fusing  data  from  these  disparate  sources  to 
visualize  a  coherent  Recognized  Maritime  Picture  (RMP)  can  be  a  challenge. 

There  may  be  a  large  number  of  contacts  in  an  AOR,  but  typically  there  are  few  if  any 
undesirable  contacts.  Operators  and  decision  makers  attempt  to  identify  undesirable  contacts 
by  looking  for  anomalous  behaviour  or  matching  contact  characteristics  with  other  intelligence 
information.  Operators  exploit  various  types  of  technology  to  collect  and  manage  data  on 
contact  behaviour  and  characteristics,  integrating  or  fusing  the  data  into  a  more  complete 
picture.  Unfortunately,  sensors  have  limited  accuracy,  may  indicate  conflicting  data  or  may 
present  irrelevant  information  due  to  sensed  marine  life  or  interference  from  the  physical 
environment. 

In  the  past,  decision  makers  have  used  training  and  experience  to  fuse  these  data  manually,  but 
this  is  a  slow  process  and  there  is  a  desire  to  automate  the  fusion  process  as  sensor  and 


1  http://www.mendelev.com/ 
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processing  technologies  improve.  The  human  operator  will,  however,  be  the  ultimate  arbitrator 
in  the  information  fusion  process  for  the  foreseeable  future.  This  raises  concerns  about  how  any 
uncertainty  that  is  associated  with  the  fused  information  should  be  presented  to  the  operator  in 
order  to  promote  understanding  and  lead  to  appropriate  decision  making. 

1 .2  Objective 

The  objective  of  this  review  is  to  better  understand  the  current  state-of-the-art  methods  for 
representing  uncertainty  in  sensor-fusion  data  using  graphical,  visual  displays  to  inform  military 
decision  making. 

The  work  is  intended  to  summarize  the  current  state  of  the  art  in  visually  representing  data- 
fusion  uncertainty,  integrating  available  information  and  interpreting  the  results  to  help  direct 
future  research  on  the  Human  Factors  element  in  developing  the  RMP  despite  inaccurate, 
incomplete  or  contradictory  data. 

1.3  Scope 

The  statement  of  work  was  focussed  on  understanding  how  to  effectively  present  the 
uncertainty  of  temporal-geospatial  data  associated  with  a  contact.  The  purpose  of  presenting 
uncertainty  is  to  enhance  understanding  of  where  the  contact  is,  has  been  and  will  be  so  that 
the  operator  is  able  to  make  better  decisions  about  the  contact’s  intent.  During  the  project  kick¬ 
off  meeting,  DRDC  and  CAE  restricted  the  scope  to  how  uncertainty  in  the  coordinates  (latitude, 
longitude  and  altitude  or  depth)  of  a  contact  that  are  derived  from  the  fusion  of  data  from 
multiple  traditional  sensors  over  a  period  of  time  can  or  should  be  indicated  on  an  electronic 
tactical  display,  particularly  in  a  Maritime  defence  and  security  context.  This  narrower  scope 
was  found  to  be  too  restrictive  during  the  literature  review  and  so  it  was  broadened  to  consider 
visualization  of  geospatial  uncertainty  associated  with  Maritime  contacts  regardless  of  the 
source  of  the  imprecision. 

This  scope  of  work  does  not  include  certain  types  of  data  uncertainty,  such  as  self-report  of 
cargo  manifest,  nor  does  it  include  of  the  inference  and  decision  making  processes  that  might 
incorporate  uncertainty.  The  automatic  fusion  algorithms  that  generate  information  presented  to 
the  operator  are  not  considered  here  other  than  to  note  that  this  processing  of  sensed  data  will 
give  rise  to  information  that  has  some  associated  uncertainty.  The  client  also  indicated  that  they 
were  specifically  interested  in  visual  representation  of  information  rather  than  alternative  forms 
such  as  auditory  or  tactile  displays,  so  the  bulk  of  the  work  is  directed  towards  uncertainty 
visualization  on  dynamic,  computer-generated,  visual  displays. 
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2  LITERATURE  REVIEW 

2.1  Method 

Searching  the  literature  was  largely  conducted  using  online,  internet  sources  using  Google  and 
Google  Scholar  search  engines  with  keywords  derived  from  the  statement  of  work.  Relevant 
documents  were  downloaded  immediately  if  available,  but  a  few  documents  were  obtained 
through  electronic  library  access  available  to  CAE  and  one  document  was  obtained  by  the  client 
through  the  DRDC  library.  Other  documents  were  retrieved  from  academic  institutions  and 
several  were  obtained  from  ResearchGate2.  Additional  references  were  retrieved  as 
recommended  by  contacted  authors  in  the  field  or  when  identified  from  citations  in  articles 
during  the  review  process. 

A  large  number  of  PDF  documents  were  identified  and  retrieved  (more  than  500).  The  retrieved 
documents  were  catalogued  using  Mendeley3  bibliographic  reference  manager  using 
Mendeley’s  document  import  capability.  This  automated  process,  while  not  perfect,  saved 
considerable  time  organizing  the  documents.  Corrections  to  the  automatically  populated  citation 
information  were  made  manually  as  the  errors  were  encountered.  The  catalogued  documents 
where  then  prioritized  for  review  as  described  below. 

2.1.1  Constraints 

The  short  duration  of  this  project  meant  that  there  was  insufficient  time  to  physically  access 
public  or  academic  library  holdings.  Only  documents  that  were  readily  and  freely  available 
electronically  were  considered;  there  was  no  budget  for  purchasing  documents  from  journals. 

2.1.2  Sources  and  Resources 

The  Defense  Technical  Information  Center4  (DTIC)  was  a  valuable  repository  of  information.  A 
number  of  documents  were  also  obtained  from  the  International  Society  of  Information  Fusion5 
(ISIF). 

Two  working  groups  were  found  that  are  relevant  to  sensor  fusion  and  its  application.  The 
Evaluation  of  Technologies  for  Uncertainty  Reasoning  Working  Group6  (ETURWG)  and  the 
Canadian  Tracking  and  Fusion  Group7  (CTFG)  both  have  active  topic  areas  involving  research 
in  the  sensor-data  visualization  domain  and  include  uncertainty  visualization  as  an  area  of 
interest. 

The  Science  and  Technology  Organization  (STO)  of  the  North  Atlantic  Treaty  Organization 
(NATO)  has  panels  with  technical  working  groups  that  are  interested  in  sensor-data 
visualization,  notably  the  following: 


2  http://www.researchgate.net/ 

3  http://www.mendelev.com/ 

4  http://www.dtic.mil/dtic/ 

5  http://www.isif.org/ 

6  http://eturwq.c4i.qmu.edu/ 

7  http://www.ctfq.ca/ 
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•  Centre  for  Maritime  Research  and  Experimentation  (CMRE) 

•  Human  Factors  and  Medicine  (HFM) 

•  System  Analysis  and  Studies  (SAS) 

•  Sensors  and  Electronics  Technology  (SET) 

Several  relevant  NATO  STO  (and  the  former  Research  and  Technology  Organization,  RTO) 
documents  were  retrieved  during  the  literature  search. 

There  are  a  few  journals  that  seem  particularly  relevant  to  the  broader  sensor-fusion 
visualization  domain: 

•  Information  Fusion:  http://www.iournals.elsevier.com/information-fusion/ 

•  Journal  of  Advances  in  Information  Fusion:  http://www.isif.org/iournal 

•  International  Journal  of  Image  and  Data  Fusion: 
http://www.tandfonline.eom/loi/tidf20#.VQs  KS4TQqA 

There  are  numerous  other  journals  that  deal  with  geo-spatial  and  temporal  visualization  as  part 
of  a  more  general  mandate,  particularly  for  researchers  interested  in  scientific  information 
visualization  (IEEE  SciVis)  or  information  visualization  (IEEE  InfoVis)  as  well  as  general 
perceptual  and  cognitive  scientific  journals. 

2.1.3  Participant  Matter  Experts 

Several  researchers  in  the  field  of  information  visualization  were  contacted  and  asked  to 
recommend  documents  relevant  to  the  literature  review. 

Dr.  M.  Riveiro8  has  published  numerous  reports  and  conference  papers  that  address  the  issue 
of  visualization  within  Maritime  defence  and  security  domain,  including  the  effects  of  uncertainty. 
Dr.  Riveiro  recommended  several  documents  for  review. 

Dr.  A.  Bisantz9  from  the  University  at  Buffalo  recommended  a  copy  of  a  similar  review  that  she 
had  conducted  for  the  Oxford  Handbook  of  Cognitive  Engineering,  which  was  obtained  by  the 
client  for  the  current  review. 

Dr.  E.  Blasch  (formerly  with  DRDC,  now  at  Wright  Patterson  AFB)  recommended  some  recent 
DRDC  reports  that  were  retrieved. 

Dr.  A-L.  Lapinski  from  DRDC  was  contacted  indicating  that  the  most  interesting  results  of 
previous  work  were  PROTECTED  B  and  not  generally  available. 


8  maria.riveiro@his.se.  http://www.his.se/om-oss/Organisation/Personalsidor/maria_riveiro/Publikationer/ 

9  bisantz@buffalo.edu.  http://www.eng.buffalo.edu/~bisantz/ 
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The  following  researchers  replied,  however,  the  topic  area  was  not  in  their  direct  area  of 
expertise  so  they  were  not  able  to  provide  specific  advice: 

•  DR.  M.  Friendly,  York  University 

•  Dr.  S.  Carpendale,  University  of  Calgary 

•  Dr.  F.  Taylor,  Carleton  University 

2.1.4  Document  Prioritization 

Two  DRDC-sponsored  reviews  of  the  visualization  literature  had  been  conducted  previously 
(Davenport  &  Risley,  2006;  Davenport,  2009)  so  preference  was  initially  given  to  more  recent 
documents  while  not  totally  excluding  earlier  works.  These  two  reviews  were  included  in  the 
current  review  as  well  as  several  other  relevant  DRDC  reports. 

The  first  pass  at  prioritizing  the  retrieved  documents  was  based  on  titles  looking  for  keywords 
and  synonyms  that  were  based  on  the  tasking  objective.  No  strict  set  of  keywords  was  used. 

The  papers  were  categorized  as  High  (242),  Medium  (131)  or  Low  (142)  “title-relevance”.  The 
242  High  title-relevance  papers  were  too  many  to  review  in  the  time  available  so  a  second 
prioritization  step  was  implemented. 

The  second  pass  reviewed  the  abstracts  of  High  title-relevance  and  15  additional  papers  to 
assess  each  paper’s  “content-relevance”.  These  were  also  categorized  as  High  (112),  Medium 
(65)  or  Low  (80).  The  112  High  content-relevance  papers  were  still  too  many  to  review  so  a 
selection  of  papers  from  the  list  was  made. 

The  selection  of  papers  to  review  was  based  on  documents  that  had  high  title  and  content- 
relevance,  focussing  on  prominent  authors  and  previous  DRDC  reports.  Particular  attention  was 
paid  to  those  documents  that  included  a  survey  of  the  literature  or  that  contained  empirical 
evaluation  of  techniques.  The  authors  that  were  highlighted  because  of  their  prominence  in  the 
literature  or  who  had  written  relevant  DRDC  reports  were:  Potter,  Pang,  Riveiro,  Blasch, 
MacEachren,  Bisantz,  Matthews  and  Lapinski. 

During  the  review  of  the  selected  papers,  additional  promising  reports  were  identified  and  added 
to  the  list  of  available  documents.  Several  of  these  documents  were  subsequently  reviewed 
resulting  in  a  final  count  of  25  papers  reviewed  in  detail  while  a  number  of  others  were  accessed 
to  verify  or  incorporate  specific  details  cited  by  other  authors. 

2.2  Literature  Review  Results 

Brown  (2004)  observed  that  there  are  a  number  of  ways  to  indicate  uncertainty  or  to  distinguish 
between  levels  of  uncertainty,  but  that  not  all  of  these  approaches  are  intuitive  metaphors  that 
convey  that  message  to  operators.  MacEachren  et  al.  (2005)  note  that  most  approaches  for 
depicting  uncertainty  in  map-based  applications  build  upon  Bertin’s  visual  variables  proposed  in 
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1983  for  traditional  cartography.  Bertin10  originally  proposed  his  visual  variables  to  depict 
attributes  of  cartographic  objects  rather  than  uncertainty  per  se,  exploiting  illustration  attributes 
such  as:  location;  size;  grain  (texture  coarseness,  which  MacEachren  differentiates  from  texture 
in  general);  colour  value;  colour  hue;  orientation;  shape.  User  interface  design  guidance 
recommends  that  symbols  used  to  represent  data  provide  a  clear,  intuitive  description  of  that 
data,  including  how  it  varies.  For  instance,  icon  size  is  an  intuitive  metaphor  for  data  magnitude; 
colour  is  a  cultural  metaphor  that  is  used  to  represent  classes  of  data  such  as  friend  or  foe.  In 
instances  where  there  is  no  explicit  mapping  between  the  visual  variable  and  the  data  the  user 
must  interpret  the  representation,  providing  opportunity  for  misinterpretation.  In  the  current 
context,  using  shades  of  colour  to  indicate  the  magnitude  of  data  uncertainty  would  not  provide 
an  explicit,  intuitive  correspondence  between  data  and  symbol  as  colour  is  unlikely  to  be 
considered  an  ordered  set  despite  its  dependency  on  light  wavelength. 

Brown  (2004),  MacEachren  et  al.  (2005)  and  a  number  of  other  authors  have  noted  that 
uncertainty  can  be  represented  by  an  intrinsic  indicator,  such  as  a  change  to  the  data  icon’s 
(also  referred  to  as  “sign-vehicles”)  shape,  colour  or  transparency.  Alternatively,  uncertainty  can 
be  represented  by  an  extrinsic  indicator,  such  as  additional  graphs,  an  associated  glyph  or  an 
overlay  that  leaves  the  primary  data  icon  unchanged,  providing  uncertainty  information 
alongside  the  primary  data. 

Riveiro  (2007)  cites  some  general  advice  provided  by  pioneers  in  information  visualization, 
notably  Tufte  and  Chambers.  Riveiro  suggests  that  designers  should  strive  for  a  high  data-to-ink 
ratio  to  convey  the  greatest  amount  of  information  in  the  smallest  space  (Tufte’s  Graphical 
Excellence)  while  ensuring  that  the  representation  does  not  perceptually  distort  the  data  (Tufte’s 
Graphical  Integrity)  by  presenting  it  in  context  and  by  applying  labels  judiciously  to  resolve 
ambiguity.  Riveiro  reiterates  Chambers  advice  to  display  information  in  a  salient  manner  using 
simple  patterns  that  draw  attention  at  a  level  commensurate  with  the  information  importance  -  in 
other  words,  design  primarily  with  the  user  in  mind,  not  the  display  technology.  Amar,  Eagan 
and  Stasko  (2005)  go  further,  observing  that  the  objective  of  the  visualization  is  not  to  display 
the  data  per  se  but  to  support  the  analysis  of  the  data.  Amar  et  al.  stress  that  there  needs  to  be 
a  much  stronger  connection  to  the  user  goals,  both  for  high-level  and  low-level  tasks,  as 
advocated  in  Annett’s  Hierarchical  Task  Analysis  (Shepherd,  2000). 

The  results  of  the  literature  review  will  be  grouped  into  intrinsic  and  extrinsic  techniques  for 
representing  data  uncertainty,  particularly  temporal-geospatial  data.  Each  of  these  categories 
has  additional  subcategories  of  methods.  Other  information  that  is  relevant  to  visualizing 
information  in  the  Maritime  defence  and  security  domain,  but  not  directly  addressing  uncertainty 
visualization,  is  presented  in  Appendix  A. 

2.2.1  Intrinsic  Techniques 

Intrinsic  data  representations  techniques  include  using  position,  size,  brightness,  texture,  colour, 
orientation,  shape,  border  (thickness,  texture  or  colour),  blurring  or  transparency.  As  previously 
noted,  intrinsic  methods  are  differentiated  from  extrinsic  representations  because  they  alter  the 
primary  data  symbol  itself,  whereas  extrinsic  representations  add  to  the  primary  data  symbol. 


10  http://www.infovis-wiki.net/index.php?title=Visual  Variables#Jaques  Bertin 
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Some  of  these  intrinsic  symbol  attributes  are  more  applicable  to  indicating  the  primary  data 
rather  than  its  uncertainty. 

The  intent  of  intrinsic  techniques  when  representing  data  uncertainty  is  largely  to  indicate  that 
some  uncertainty  with  the  primary  data  is  present  rather  than  the  magnitude  of  the  uncertainty, 
although  some  authors  have  attempted  to  do  both  as  will  be  described  below.  In  the  specific 
example  of  contact  location  icons,  intrinsic  symbol  modifications  would  not  directly  indicate  the 
extent  positional  uncertainty  beyond  the  nominal  symbol  location.  This  is  often  done  to  reduce 
clutter  that  would  obscure  other  map  features  or  symbols.  MacEachren  et  al.  (2005)  note  that 
colour  hue,  colour  value  and  texture  are  common  methods  of  indicating  uncertainty  of  static 
data  in  cartography. 

2.2. 1.1  Transparency  and  Saturation 

MacEachren  et  al.  (2005)  summarize  three  intrinsic  methods  for  depicting  point  data  uncertainty 
(such  as  the  location  of  a  vessel  on  a  map.)  A  reproduction  of  the  approaches  is  presented  in 
Figure  2-1 ,  shown  with  uncertainty  increasing  from  left  to  right.  The  top  scheme,  Colour 
saturation,  attempts  to  depict  increasing  uncertainty  by  reducing  the  saturation  level  of  the 
symbol  fill,  achieved  by  increasing  the  gray  content  of  the  symbol  fill  colour.  The  middle 
scheme,  Contour  crispness,  attempts  to  depict  increasing  uncertainty  by  reducing  the  crispness 
of  the  symbol  boundary.  The  reproduction  shown  here  is  a  poor  approximation  of  the  original 
figure  with  fuzziness  replaced  by  concentric  circles  of  lighter  shades  of  fill.  The  bottom  scheme 
uses  MacEachren’s  fog  metaphor  where  the  increasing  opaqueness  of  the  inner  circle 
corresponds  to  greater  uncertainty.  In  MacEachren’s  technique  shown  in  Figure  2-1,  symbol 
size  does  not  change  with  uncertainty,  so  the  intrinsic  visualization  concept  shown  here  is 
distinctly  different  from  extrinsic  techniques  such  as  the  threat-rings  used  in  current  CAF 
Maritime  tactical  displays. 


Colour  saturation 
Contour  crispness 
Transparency 


0  O  • 


Figure  2-1:  Location  Uncertainty  Indication.  Symbol  uncertainty  increases  from  left  to 

right. 

Image  redrawn  based  on  the  original  image  in  MacEachren  et  al.  (2005) 


MacEachren  et  al.  (2005)  go  on  to  describe  some  other  authors’  approaches,  such  as  Drecki’s 
icon  opaqueness,  where  increasing  uncertainty  was  depicted  by  increasing  opaqueness,  which 
would  obscure  detail  under  the  icon.  Another  is  Pang’s  (2001)  use  of  glyphs  for  visualizing 
uncertainty  within  flow  fields.  Although  Pang’s  application  was  extrinsic,  the  concept  is 
extensible  to  a  discrete  object  symbol,  where  the  base  icon  shape  is  distorted  or  augmented 
(forming  a  compound  symbol)  to  denote  uncertainty.  The  use  of  visualization  in  distributed, 
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continuous  phenomena  (such  as  land  use,  flow  fields,  weather,  etc.)  is  a  common  technique  for 
researchers  using  geospatial  data  (population  distribution,  agriculture,  weather,  et  cetera.) 

Bisantz  (2013)  notes  that  different  colours  can  be  used  as  primary  data  indicators,  such  as 
Pfeifer’s  use  of  colours  to  visualize  robotic  machine  learning.  Pfeifer  represented  different 
classes  of  objects  by  different  colours,  as  is  also  done  in  CAF  tactical  plots,  and  then  used 
brightness  to  indicate  belief  about  the  certainty  of  an  object’s  location  and  saturation  to 
represent  the  certainty  about  the  object’s  presence. 

Bisantz  (Bisantz,  Marsiglio,  &  Munch,  2005;  Bisantz,  2013;  Finger  &  Bisantz,  2000)  reported  on 
a  series  of  studies  using  a  symbol-fill  strategy  to  indicate  object  classification  uncertainty.  They 
compared  the  fill  techniques  to  linguistic  indications  (numerical  percentages,  descriptive  verbal 
labels)  and  an  arrow  icon  with  varying  sharpness.  They  applied  textures  comprising  randomly 
distributed  mixtures  of  red  and  green  squares  where  the  proportions  of  each  colour  were 
proportional  to  the  probabilities  that  the  contact  belonged  to  each  of  two  categories.  The  results 
indicated  that  when  the  graphical  icon  intrinsic  properties  were  degraded  proportionally  to  the 
uncertainty,  decision  makers  were  able  to  make  use  of  that  information. 

Bisantz  (2013)  cites  one  of  her  earlier  studies  (Bisantz  et  al.,  2009)  that  indicated  participants 
could  use  transparency,  saturation  and  brightness  to  rank  order  symbols  but  that  hue  was  not  a 
reliable  indicator.  In  this  same  study,  Bisantz  used  the  brightness  of  the  symbol  to  characterize 
the  recency  of  the  data;  however,  participants  became  confused  about  the  correspondence 
between  darkness  of  the  icon  and  recency  of  the  information,  with  some  participants  adopting 
opposite  interpretations  from  other  participants.  A  follow  up  study  suggested  that  participants 
were  significantly  influenced  by  contrast  between  the  icon  colour  and  the  display  background 
than  by  the  absolute  level  of  saturation  of  the  symbol  colour. 

Bisantz  (2013)  cites  another  of  her  studies  (Bisantz  et  al.,  2011)  in  which  transparency  was 
used  to  graphically  indicate  uncertainty  about  objects  both  with  and  without  explicit  indication  of 
the  associated  numerical  estimate  of  the  uncertainty  (see  Figure  2-2).  Participants  seemed  to 
adopt  different  strategies  when  transparency  was  used  even  if  the  numerical  value  was  also 
presented.  Better  scores  were  obtained  when  object  uncertainty  was  indicated  using 
transparency  regardless  of  whether  the  numerical  probability  was  presented.  Mental  workload 
did  not  vary  significantly  with  display  conditions  (although  it  was  somewhat  higher  when 
participants  could  toggle  the  display  of  uncertainty  information,  presumably  due  to  the  additional 
interaction  with  the  display  when  hiding  or  revealing  information.)  Participants  reacted  sooner 
and  with  less  evidence  when  they  were  presented  with  the  uncertainty  information  graphically 
compared  to  when  information  was  only  presented  as  the  primary  data  icon  that  could  only 
change  to  indicate  the  highest  probability  contact.  Presenting  the  associated  numerical 
uncertainty  value  affected  the  participant  performance  somewhat,  leading  to  more  conservative 
decisions  than  when  the  numerical  value  was  not  presented. 
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Icon  and  numerical  Icon  without  numerical 

uncertainty  probability  uncertainty  probability 


Control  condition: 

Icon  varied  only  between  type 
of  object,  not  transparency 
(aircraft  is  the  most  likely 
identification) 


Experimental  condition: 

Icon  transparency  dependent 
upon  uncertainty  (aircraft  is  the 
most  likely  identification) 


Experimental  condition: 

Icon  transparency  dependent 
upon  uncertainty  (missile  is  the 
most  likely  identification) 


Figure  2-2:  Example  of  using  icon  transparency  (on  a  gray  background)  to  demonstrate 
uncertainty  for  2  object  types  both  with  and  without  numerical  uncertainty. 

Image  redrawn  based  on  the  original  image  in  Bisantz  et  al.  (2011). 

Riveiro  (2007)  cites  some  authors11  who  offer  the  opinion  that  varying  colour  saturation  is  not  an 
effective  means  of  depicting  uncertainty  and  others12  who  feel  that  the  evidence  supports 
lightness  to  be  the  more  intuitive  representation  method  when  using  intrinsic  colour  attributes  to 
indicate  uncertainty  magnitude.  However,  it  is  unclear  whether  the  differences  in  opinion  among 
researchers  stem  from  differences  between  studies  or  if  the  benefits  of  visualizing  uncertainty 
differ  so  much  between  tasks  that  generalizing  observations  to  other  contexts  may  be 
premature,  requiring  additional,  qualifying  constraints. 


11  M.  Leitner  and  B.  P.  Buttenfield,  “Guidelines  for  the  display  of  attribute  certainty,”  Cartography  and 
Geographic  Information  Science,  vol.  27,  no.  1,  pp.  3-14,  2000. 

Schweizer  and  Goodchild  (D.  Schweizer  and  M.  Goodchild,  “Data  quality  and  choropleth  maps:  An  experiment 
with  the  use  of  color,”  in  Proceedings,  GIS/LIS,  vol.  2,  1992,  pp.  686-699. 

12  B.  A.  Jiang  B.  and  F.  J.  Ormeling,  “Some  perceptual  aspects  of  colouring  uncertainty,”  Advances  in  GIS 
Research  II,  pp.  477-490, 1996 
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2.2. 1.2  Blurriness  and  Animation 

As  noted  in  the  preceding  section,  the  use  of  visualization  in  distributed,  continuous  phenomena 
(such  as  land  use,  flow  fields,  weather,  etc.)  is  a  common  technique  for  researchers  using 
geospatial  data.  MacEachren  notes  that  authors  have  used  animation,  such  as  rapidly  changing 
colours,  to  indicate  uncertainty. 

Bisantz  (2013)  cites  studies  of  icon  blurriness  (Bisantz  et  al.,  2005;  Finger  &  Bisantz,  2000)  that 
indicate  participants  are  capable  of  distinguishing  and  ordering  images  among  a  sequential 
range  of  distorted  icons.  Finger  and  Bisantz  (2000)  investigated  the  use  of  icon  pairs  that  had 
culturally  semantic  connotations  as  well  as  abstract  figures  but  did  not  find  any  significant  effect 
during  identification  or  rank  ordering  performance.  In  the  culturally  semantic,  iconic  pair 
condition,  they  used  either  a  (happy-face,  sad-face)  or  a  (dove-of-peace,  skull-and-crossbones) 
icon  pair  to  indicate  (friendly,  hostile)  objects.  In  the  abstract  pair  condition  they  used  either  (Pi, 
A)  or  (O,  X)  symbols  for  (friendly,  hostile)  objects.  They  used  squares  coloured  either  green  or 
red  to  represent  friendly  or  hostile  objects  respectively  in  the  combined  abstract-iconic  condition. 

Finger  and  Bisantz  (2000)  observed  that  the  time  to  identify  an  object  did  not  depend  upon  on 
the  type  of  icon  used,  suggesting  that  participants  quickly  learned  to  associate  meaning  even  to 
abstract  symbols,  although  there  were  only  two  categories  of  objects  (i.e.  friendly  or  hostile). 

The  subsequent  studies  reported  in  Bisantz  et  al.  (2005)  extended  the  investigation  of  Finger 
and  Bisantz  by  also  including  colour  texture  variability,  verbal  uncertainty  labels  and  numerical 
estimates  of  uncertainty  in  a  decision  making  task.  Performance  was  found  to  be  largely 
unaffected  by  the  method  of  representing  uncertainty  although  there  was  an  interaction  whereby 
decision  time  decreased  in  the  higher  uncertainty  conditions  when  some  indication  of  the 
uncertainty  was  presented. 

Brown  (2004)  proposes  the  use  of  animation  in  the  form  of  vibrating  features  to  convey  the 
concept  of  temporal  uncertainty.  Brown  suggests  that  blurring  is  an  effective  metaphor  when 
trying  to  depict  uncertainty,  achieved  by  retaining  many  of  the  gross  object  features  but 
eliminating  finer  features  and  smearing  the  corresponding  details.  By  introducing  a  vibrational 
component  to  objects,  the  image  features  can  oscillate  to  indicate  uncertainty  of  location  over 
an  area.  Brown  notes  that  Tufte  advises  against  using  vibration  in  displays  as  it  can  be 
annoying  to  the  operator  and  this  is  reiterated  in  MacEachren  et  al.  (2005).  However,  Brown 
feels  that  there  may  be  applications  were  it  is  appropriate,  particularly  when  it  is  important  to 
attract  the  operator’s  attention  to  the  fact  that  some  aspect  is  uncertain  or  requires  further 
inspection. 

Brown  (2004)  indicates  that  oscillating  between  colours  or  varying  intensity  are  effective  means 
of  demonstrating  dynamic  visual  effects,  even  with  stationary  objects.  Vibrating  textures,  high 
frequency  changes  in  contrast,  hues,  luminance,  etc.,  can  be  used  to  create  a  shimmering  effect 
to  highlight  that  there  is  some  uncertainty.  Such  an  approach  is  probably  not  effective  at 
denoting  the  magnitude  of  the  uncertainty,  but  should  be  effective  for  indicating  categorical 
ranges  (e.g.  high,  medium,  low).  Brown  discusses  this  approach  in  the  context  of  terrain  maps, 
but  similar  methods  could  be  conceived  for  object  temporal-spatial  uncertainty.  Animation 
approaches  can  be  applied  to  object  fill  or  to  border  shape. 
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2.2.2  Extrinsic  Techniques 

Extrinsic  techniques  of  representing  information  are  approaches  that  leave  the  primary  data  icon 
unchanged  but  add  to  it,  creating  a  composite  image.  For  example  dials,  thermometers,  arrows, 
bars,  pie  charts,  graphs,  complex  objects,  glyphs,  et  cetera  may  be  used  to  display  additional 
information  or  to  represent  that  additional  information  is  available.  Similar  to  intrinsic  techniques, 
some  extrinsic  techniques  are  better  suited  for  representing  uncertainty  than  others. 

2.2.2.1  Descriptive  Labels 

Descriptive  labels  are  added  alpha-numeric  notations  associated  with  the  primary  data  icon  that 
provide  an  indication  of  either  the  numerical  value  or  a  verbal,  linguistic  description  of  an 
contact’s  uncertainty.  The  studies  of  Bisantz  et  al.  (2005)  reported  in  section  2. 2. 1.2  also 
included  extrinsic  representations  of  uncertainty,  labelling  icons  with  either  a  numerical 
probability  or  a  corresponding  verbal  descriptor  (e.g.  ‘Very  unlikely’,  ‘Uncertain’,  ‘Better  than 
even’,  et  cetera.)  Performance  in  a  decision  making  task  did  not  depend  substantially  on 
whether  uncertainty  was  indicated  extrinsically  or  embodied  in  the  icon  intrinsically. 

2.2.2.2  Animation 

Brown  (2004)  cites  MacEachren  (1992)  who  suggests  using  iconic  pairs  for  representing  object 
information:  a  static  icon  for  the  primary  data  coupled  with  a  secondary  icon  for  secondary  data 
(e.g.  uncertainty)  that  may  be  animated.  Brown  also  cites  Kardos  et  al.  (2006)  who  applied 
blinking  polygons  and  colours  (among  other  techniques)  to  highlight  uncertainty  in  area  data 
shown  on  a  map.  The  results  suggest  that  blinking  was  the  preferred  technique  for  conveying 
uncertainty  in  the  task  studied;  however,  this  preference  may  be  significantly  dependent  upon 
the  task,  notably  the  number  of  simultaneously  blinking  objects  and  the  task  duration. 

2. 2.2. 3  Glyphs 

Pang,  Whittenbank  and  colleagues  (A.  T.  Pang,  Wittenbrink,  &  Lodha,  1997;  Wittenbrink, 

Saxon,  Furman,  Pang,  &  Lodha,  1996)  define  a  glyph  as  “...a  geometrically  plotted  specifier  that 
encodes  data  values”,  although  a  glyph  could  equally  well  be  described  more  generally  as  an 
icon  or  symbol  used  to  convey  meaning.  We  will  primarily  focus  on  extrinsic  glyphs  in  this 
section,  although  the  definition  and  the  examples  overlap  with  intrinsic  methods,  particularly 
varying  shape  or  colour.  However,  a  glyph  can  also  be  an  addition  of  a  symbol  to  the  primary 
data  icon,  such  as  the  threat  rings  that  are  currently  applied  to  contact  icons  on  tactical  plots.  In 
one  of  the  examples  offered,  arrow  glyphs  were  used  to  indicate  ocean  current  primary  data  of 
direction  (arrow  direction)  and  speed  (arrow  length).  They  encoded  uncertainty  of  these  primary 
data  by  varying  the  width  of  the  arrows  and  the  number  of  arrow  heads.  They  note  that  thicker 
(uncertain)  arrows  are  more  salient,  which  may  divert  attention  inappropriately,  but  that  intrinsic 
variation  of  colour  brightness  may  be  used  to  counterbalance  this  effect. 

Bisantz  (2013)  notes  that  numerous  researchers  have  used  rings  or  ellipses  to  characterize 
location  uncertainty,  notably  the  pioneering  work  by  Andre  and  Cutler  (1998).  This  technique  is 
similar  to  the  use  of  contact  threat  rings  or  submarine  location  uncertainty  ellipses  in  current 
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tactical  displays.  Andre  and  Cutler  (1998)  studied  three  approaches  for  depicting  location 
uncertainty  and  three  approaches  for  depicting  heading  uncertainty  as  shown  in  Figure  2-3. 


A  $  I 


(a)  (b) 

Figure  2-3:  Extrinsically  depicting  uncertainty  associated  with  the  primary  data. 

(a)  Increasingly  uncertain  location  from  left  to  right;  (b)  Various  heading  indicators  with  a 
single  uncertainty  value  of  30  degrees.  The  location  uncertainty  colour  coding  method 
(lower  row  of  figure  a)  is  actually  classified  as  intrinsic. 

Images  redrawn  from  descriptions  and  original  images  in  (Andre  &  Cutler,  1998). 

In  Andre  and  Cutler’s  (1998)  navigation-task  experiment,  participants  had  to  navigate  an  icon 
representing  their  own-ship  around  a  moving  object  to  avoid  collision  while  minimizing  course 
deviation.  The  nominal  location  of  the  object  was  indicated  by  a  point-location  icon  that  was 
common  to  all  conditions.  The  uncertainty  of  the  location  of  the  moving  object  was  indicated  in 
one  of  three  ways,  two  graphical  and  one  textual  as  illustrated  in  Figure  2-3a.  The  colour  coding 
method  (lower  row  of  Figure  2-3a)  is  actually  classified  as  intrinsic. 

In  this  first  experiment,  they  observed  that  although  representations  produced  fewer  collisions 
than  the  control  condition  (point  icon  only),  the  explicit  representation  of  the  circular  area  of 
possibility  resulted  in  the  fewest  collisions. 

In  Andre  and  Cutler’s  (1998)  second  experiment,  they  used  simple  triangular  icons  with  a 
heading  vector  to  indicate  contacts,  similar  to  tactical  plots.  Participants  were  asked  to  judge 
hostile  intent  based  on  approach  behaviour.  Uncertainty  in  a  contact’s  heading  was  indicated 
extrinsically,  either  numerically  or  with  one  of  two  graphical  arc  representations  as  shown  in 
Figure  2-3:  Extrinsically  depicting  uncertainty  associated  with  the  primary  data. 

They  observed  that  all  of  the  heading  uncertainty  indication  methods  were  superior  to  the 
control  condition  of  no  heading  uncertainty  indication,  but  differences  among  the  methods  were 
minor. 

Bisantz  (2013)  cites  Kirschenbaum  and  Arruda  (1994)  who  studied  the  display  of  uncertainty  in 
a  simulated  submarine  hunting  experiment  using  ellipses  or  verbal  descriptions  to  indicate 
uncertainty  of  the  relative  submarine  location  as  it  moved  across  the  map  display.  The  control 
condition  showed  only  the  nominal  submarine  track  on  the  display.  Both  of  the  uncertainty 
conditions  were  superior  to  the  control  condition  when  predicting  range,  but  there  was  no 
significant  difference  between  the  uncertainty  condition  results.  Bisantz  (2013)  also  notes  that 
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colour  brightness  has  been  used  by  Slocum  and  colleagues  to  map  water  availability,  with  bright 
coloured  areas  being  more  certain  than  dark  coloured  areas.  Such  an  approach  could  also  be 
applied  to  area  glyphs  to  denote  locational  uncertainty,  similar  to  that  used  in  submarine 
location  uncertainty  ellipses  currently  used  on  tactical  displays. 

In  a  subsequent  experiment,  Kirschenbaum  et  al.  (2013)  manipulated  uncertainty  displays, 
presenting  the  uncertainty  of  the  nominal  primary  data 

•  numerically,  with  uncertainty  presented  in  a  table  above  a  graphical  plot  of  the  nominal 
object  location 

•  graphically,  showing  spatial  extent  of  the  probably  nominal  object  location  on  the  graphical 
plot 

•  a  combination  of  tabular  and  graphical  indications  of  uncertainty  of  the  nominal  object 
location 

Kirshenbaum  et  al.  (2013)  found  that  experts  were  more  accurate  than  novices  when  the 
uncertainty  information  was  presented  numerically  in  the  table,  but  displaying  the  location 
uncertainty  graphically  improved  performance,  allowing  novice  performance  to  approach  that  of 
the  experts.  There  was  no  significant  difference  for  the  time  to  make  a  decision  in  any  of  the 
conditions. 
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Bisantz  (2013)  cites  Schaefer  et  al.  (2004)  who  used  an  Air  Traffic  Control  (ATC)  scenario  to 
assess  the  effectiveness  of  uncertainty  bubble  overlays  for  two  prototype  decision  support  tools 
being  investigated  by  Eurocontrol.  The  Potential  Problem  Display  (PPD)  displays  predicted 
conflict  between  aircraft  in  a  horizontal,  plan  view  while  the  Vertical  Assistance  Window  (VAW), 
which  displays  vertical  flight  profiles  (see  Figure  2-4).  These  tools  were  presented  in  addition  to 
the  traditional  plan  view  of  the  controller’s  assign  airspace  responsibilities.  They  observed  that 
ATC  operators  found  the  concept  of  uncertainty  foreign  and  the  prevailing  opinion  was  that 
displaying  uncertainty  required  extra  effort  while  not  providing  much  benefit. 


Figure  2-4:  PPD  and  VAW  from  FACTS  1b  Trial. 
Reproduced  from  Whysall  (1998).  ©  EUROCONTROL 


Riveiro  et  al.  (2014)  conducted  a  study  using  a  simplified  tactical  plot  in  a  Maritime  defence  and 
security  decision  making  scenario  with  military  ATC  officers.  The  control  condition  contained  no 
indication  of  data  uncertainty,  although  the  uncertainty  was  randomly  manipulated  during  all  the 
trials.  In  the  control  condition,  contact  state  data  for  altitude,  bearing,  range  and  speed  were 
presented  for  the  selected  contact  in  graphs  as  a  function  of  time-late,  each  similar  to  the  top 
graph  on  the  left  side  of  Figure  2-5.  Geographical  location  with  a  velocity  vector  was  presented 
on  a  map  display  for  each  contact,  similar  to  single  contact  shown  on  the  right  side  of  Figure  2-5 
but  with  only  the  nominal  location,  not  the  uncertainty  circle.  In  the  experimental  group, 
uncertainty  for  the  contact  state  data  was  presented  as  in  the  lower  left  graph  of  Figure  2-5 
while  the  tactical  plot  was  as  shown  on  the  right. 
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Figure  2-5:  Examples  of  data  presentation  for  a  Maritime  ATC  task. 

Image  redrawn  based  on  original  images  in  Riveiro  et  al.  (2014) 

The  results  did  not  indicate  any  significant  difference  in  the  time  it  took  to  make  decisions  or  in 
the  accuracy  of  those  decisions.  Participant  confidence  was  also  similar  between  groups.  The 
authors  note  that  null  effects  such  as  this  are  common  for  experiments  involving  decision 
making  with  uncertainty  displays  and  cite  other  studies  available  in  the  literature.  The  results  did 
not  confirm  a  study  by  Stone  and  colleagues13  who  observed  participants’  behaviour  was  more 
conservative  when  uncertainty  was  displayed. 

In  a  subsequent  report  (Riveiro,  Helldin,  &  Falkman,  2014)  extended  previous  work,  using 
fighter  pilots  making  judgements  about  contacts  employing  an  automated  decision-aid  that 
displayed  information  fusing  recommendations.  The  experimental  conditions  included  a  control 
(textual  data  with  no  automated  fusion)  and  two  experimental  conditions  (automatic  contact 
classification  displayed  textually;  textual  classification  and  graphical  values  of  the  sensor  data.) 
Both  experimental  conditions  significantly  increased  decision  time,  but  there  was  no  significant 
difference  for  incorrect  classifications.  Although  there  were  minor,  insignificant  differences  in  the 
classification  and  trust  ratings,  participant  opinions  suggested  that  the  additional  information 
would  have  led  to  greater  trust  and  faster  decisions  if  they  had  more  opportunity  to  train  with  the 
experimental  displays,  so  learning  effects  likely  confounded  the  manipulations  and  the 
conclusions  should  be  suspect. 

Lapinski  (2010)  reports  on  studies  conducted  by  HumanSystems  Inc.  and  DRDC  that  assessed 
two  different  approaches  for  data  quality  representations.  The  first  was  carried  out  by  Matthews 
et  al.  (Matthews,  Rehak,  Famewo,  Taylor,  &  Robson,  2008)  and  used  colour  (identity  indication) 


13  Stone,  E.R.,  Yates,  J.F.  &  Parker,  A.M.  (1997)  Effects  of  numerical  and  graphical  displays  on  professed  risk¬ 
taking  behavior.  Journal  of  Experimental  Psychology:  Applied.  3(4)  pp.243-256 
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and  geometry  (location  and  time)  glyphs  to  indicate  the  precision  of  the  primary  data  as  shown 
in  Figure  2-6.  Spatial  uncertainty  was  coded  by  the  number  of  squares  in  the  horizontal  bar 
(Lego  design)  or  the  size  of  the  black  dot  (Rectangle  design)  -  the  smaller  the  number  of 
squares  or  circles,  the  more  precise  the  location  was  known.  Time-late  uncertainty  was  coded 
by  the  location  of  the  black  dot  in  the  rectangle  (Rectangle  design)  or  by  the  number  of  squares 
in  the  vertical  bar  (Lego  design)  -  more  recent  data  was  indicated  by  positioning  the  black  dot  to 
the  left  and  moving  it  to  the  right  to  indicate  older  data  while  the  number  of  vertical  rectangles 
corresponded  to  the  age  of  the  data. 
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Figure  2-6:  Two  methods  using  colour  and  geometry  in  glyphs  added  to  the  primary  data 

icon  to  display  uncertainty  in  the  primary  data. 

Image  reproduced  from  (Matthews  et  al.,  2008). 

Analysis  of  these  glyphs  in  a  search  task  indicated  a  small  but  significant  advantage  for  search 
time  with  the  Rectangle  representation  compared  with  the  Lego  representation  but  both 
methods  resulted  in  similar,  low  error  levels  (Matthews  et  al.,  2008). 

The  second  approach  provided  indication  of  the  geographic  extent  of  the  sensor  coverage 
uncertainty  as  shown  in  Figure  2-7.  Although  this  technique  was  studied  as  an  indication  of 
sensor  coverage,  a  similar  concept  could  be  applied  as  a  secondary  contact-glyph  to  indicate 
the  extent  of  associated  uncertainty  of  location  and  time-late.  In  this  approach,  light  gray  or  fine 
grid  fill  indicated  good  quality  sensor  coverage  while  dark  gray  or  coarse  grid  fill  indicated  poor 
sensor  quality.  Time-late  was  indicated  by  the  border  attributes,  either  by  colour  (with  gold 
indicating  old  data  and  green  indicating  current)  or  by  style  (with  broken  indicating  old  and  solid 
indicating  current). 
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Figure  2-7:  Two  graphical  methods  of  indicating  sensor  coverage  and  quality. 
Image  reproduced  from  (Lapinski,  2010). 

Addition  of  the  sensor  coverage  indicators  to  the  plots  with  the  contact-glyphs  of  Figure  2-6  did 
not  result  in  a  significant  improvement  in  search  time.  The  difference  in  search  times  between 
conditions  with  the  two  sensor  coverage  approaches  was  small  (0.3  to  0.4  seconds)  but 
statistically  significant. 

McFadden  et  al.  (2008)  cite  earlier  work14  using  glyphs  shown  in  Figure  2-8,  however  the  earlier 
work  only  considered  SME  preference.  McFadden  et  al.  used  standard  primary  data  symbols 
and  applied  one  of  two  extrinsic  glyphs  (or  alternatively  adjusted  the  primary  data  symbol 
intrinsic  colour  saturation)  to  indicate  data  quality  as  shown  in  Figure  2-9. 


14 

Lockheed  Martin  Canada  Inc.  (2001).  COMDAT  uncertainty  trade  off  analysis.  Lockheed  Martin  Canada  Inc., 
Montreal,  PQ.  (Power  Point  Presentation). 

09  June  2015  -17-  5769-001  Version  03 


©  Her  Majesty  the  Queen  in  Right  of  Canada,  as  represented  by  the  Minister  of  National  Defence,  2015 


CAE 


Visualizing  Uncertainty  for  Data  Fusion  Graphics 
Review  of  Selected  Literature  and  Industry  Approaches 


a)  Slider  (Also  known  ar,.  djraining  bucket) 


oilOBioa 


b)  Sights  (Also  known  as  gun  sights) 


c)Baare. 


d)  Dote 


III 


Figure  2-8:  Secondary  glyphs  used  to  indicate  data  quality  assessed  by  Lockheed  Martin. 

Image  reproduced  from  (McFadden  et  al.,  2008) 
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Figure  2-9:  Two  extrinsic  and  one  intrinsic  methods  to  indicate  data  quality. 
Image  reproduced  from  (McFadden  et  al.,  2008). 
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McFadden  et  al.  found  that  the  intrinsic,  saturation  method  resulted  in  longer  search  times  and 
poorer  accuracy  than  the  extrinsic  methods.  The  extrinsic  methods  also  increased  the  search 
time  relative  to  the  control  condition  with  no  uncertainty  indication.  In  the  discussion,  McFadden 
et  al.  indicate  that  the  bar  method  seemed  to  be  the  most  successful  of  the  three  methods 
studied,  but  that  it  did  result  in  some  response  inconsistencies  so  there  may  be  difficulty 
differentiating  among  the  uncertainty  level  indications.  The  ring  method  was  found  to  be  the 
least  successful,  significantly  interfering  with  searching,  although  the  authors  speculate  that  this 
may  be  due  to  an  experimental  confound  of  confusion  with  range-rings  used  on  military  displays 
(the  participants  were  military  personnel.) 

2.2.2.4  Track  History  and  Projection 

Moving  contacts  add  the  dimension  “time”  that  may  be  relevant  to  the  decision  maker,  carrying 
geospatial  uncertainty  over  an  area  rather  than  at  a  single  position.  An  example  from  the  civilian 
domain  is  predicting  the  track  of  hurricanes,  as  in  Figure  2-10,  when  planning  relief  efforts. 


Warning 


<34kts>  <&4kts>  4  Hir>d  Probabilities  [ 

*  a**.,.*  TaI  ™  *9 


Figure  2-10:  Prediction  of  the  potential  track  of  tropical  storm  Arthur,  2014. 
Image  retrieved  and  reproduced  from  https://sky7weather.wordpress.com/ 

Another  aspect  relevant  to  Maritime  defence  and  security  is  the  visualization  of  effects  that 
currents  and  winds  may  have  on  the  location  of  vessels  in  distress  during  Search  and  Rescue 
(SAR)  operations.  This  suggests  that  some  technique  for  visualizing  spatial  uncertainty  of  data 
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fused  over  time  would  be  as  advantageous  as  that  from  fusing  data  from  multiple  sensors  for  a 
point  location. 

Montello  (2009)  notes  that  without  a  detailed  understanding  of  the  expert  operator’s  tacit 
knowledge,  designers  can  fail  to  adequately  support  the  operator.  They  provide  an  example 
where,  when  displaying  route  information,  a  designer  may  only  show  nominal  shipping  lanes  or 
sea  bed  topology  but  may  fail  to  display  both;  however,  a  ship’s  captain  may  choose  a  route  that 
safely  minimizes  distance  traveled  yet  is  outside  of  the  normal  shipping  lanes.  Providing  the 
operator  with  both  the  shipping  channels  and  the  seabed  topology  may  be  essential  to 
determining  whether  the  ship  is  operating  legally,  if  somewhat  anomalous,  or  is  pursuing  illegal 
activities.  The  operator  may  need  to  be  aware  of  the  rationale  behind  such  observable 
behaviours,  such  as  a  captain’s  desire  to  minimize  sailing  distance,  to  determine  whether  a 
track  history  demonstrates  anomalous  behaviour  is  reasonable  and  thus  probably  legitimate. 

Bisantz  (2013)  cites  Clausner  and  Fox  (2005)  who  created  a  set  of  visual  metaphors  for 
depicting  temporal  uncertainty  of  geospatial  data,  although  no  assessment  of  their  effectiveness 
was  made.  The  shapes  consisted  of  regular  shapes  such  as  ovals  and  rectangles  oriented 
along  track  lines.  One  example  of  the  visual  metaphors  is  reproduced  in  Figure  2-1 1 ,  which, 
while  originally  intended  to  indicate  only  temporal  uncertainty  along  a  timeline,  might  be  adapted 
to  also  indicate  spatial  uncertainty  by  varying  the  oval  width.  In  Clausner  and  Fox’s  metaphor, 
solid,  linear  objects  (lines  and  rectangles)  represent  certainty  while  dashed  and  rounded  objects 
(ovals)  denote  uncertainty.  The  vertical  lines  representing  the  temporal  extent  would  be  omitted 
if  there  was  uncertainty  about  the  start  or  end  times.  The  solid  oval  denotes  a  fixed  event  of 
short  duration  while  the  dashed  oval  occurs  over  a  longer  interval  and  may  slide  along  the 
timeline  due  to  uncertainty  in  its  specific  duration. 


Figure  2-11:  Sample  visual  metaphors  for  events  happening  in  time. 

Image  redrawn  based  on  original  image  in  Clausner  and  Fox  (2005). 

2.2.3  User  Control 

Several  authors  advocated  some  form  of  user  control  over  when  and  how  uncertainty  is 
displayed.  Some  of  this  may  simply  be  repetition  of  “common  sense”  but  the  opinions  expressed 
by  ATC  officers  reported  by  Schaefer  et  al.  (2004)  supports  this  perspective.  While 
accommodating  uncertainty  in  decision  making  may  be  culturally  constrained,  the  underlying 
human  factors  of  detection  among  distractors  and  information  overload  suggests  that  operator 
control  will  be  an  important  factor  to  consider  to  ensure  comprehension  (Fewell  &  Hazen,  2005; 
Nicholls  &  Battino,  2003;  Vatin  &  Napoli,  2013). 

Riveiro  et  al.  (2014)  and  Gomez  &  Chimento  (2011)  note  that  many  decision  making  aids  fail  to 
provide  information  that  is  the  basis  for  recommendations.  They  recommend  that  data  fission 
tools  be  provided  to  allow  the  operator  to  deconstruct  the  fused  information  to  expose  the 
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contributing  data  and  better  understand  what  the  fused  information  is  indicating.  This  may  be 
particularly  true  with  high  level  information  fusion,  but  as  the  number  of  sensors  increases  and 
as  fusion  algorithms  become  more  comprehensive,  operators  will  need  some  means  to  verify 
and  establish  trust  or  recognize  when  the  automation  is  unreliable. 

Bisantz  et  al.  (201 1)  found  that  participants  performed  better  when  they  could  toggle  between 
displaying  and  hiding  uncertainty  information.  Participants  seemed  to  be  using  the  visualization 
or  uncertainty  but  not  the  numerical  indication  of  uncertainty  as  indicated  by  higher  performance 
scores  for  the  graphical  uncertainty  condition.  The  improved  performance  came  with  a  small 
cost  of  slightly  greater  mental  workload,  principally  in  the  Physical  Demand  category  of  the 
NASA  TLX  rating  scale. 

As  noted  previously,  Amar  and  colleagues  (Amar  et  al.,  2005;  Amar  &  Stasko,  2004) 
recommend  a  strong  focus  on  user-centric  design  with  “analytic  primacy”  as  the  main 
consideration  driving  visualization  designs  rather  than  a  data-centric  “representation  primacy” 
perspective.  They  challenge  Tufte’s  admonishment  to  “above  all  else,  show  the  data”  that,  in 
some  respects,  is  counter  to  the  objective  of  data  fusion,  depending  upon  what  is  considered  as 
data.  Amar  et  al.  feel  that  this  “representation  primacy”  will  not  adequately  support  interpretation 
but  caution  that  this  does  not  mean  a  black-box  approach  (to  data  fusion)  should  be  adopted. 
The  operator  should  be  provided  with  “discovery  tools15”,  allowing  a  “white  box”  approach  that 
supports  effective  operator  interaction  with  the  presented  information  to  expose  uncertainty 
what  is  presented  and  provide  a  means  to  understand  the  implications  of  the  uncertainty 
specifics. 

2.2.4  Literature  Review  Discussion 

The  literature  of  uncertainty  visualization  and  the  related  field  of  decision  making  under 
uncertainty  are  multidisciplinary  and  extensive  (MacEachren  et  al.,  2005).  The  literature  on 
multisensory  data  fusion  is  somewhat  smaller,  but  still  large  judging  from  the  number  of 
references  returned  from  Google  searches.  Restricting  a  literature  review  to  only  Maritime 
defence  and  security  applications  makes  the  task  more  manageable,  but  may  overlook  many 
useful  papers  that  deal  with  visualizing  uncertainty  for  geospatial  data  generally  and  other 
military  applications  specifically. 

MacEachren  (2005)  notes  that " ...(l)information  uncertainty  is  a  complex  concept  with  many 
interpretations  across  knowledge  domains  and  application  contexts."  Although  the  current 
tasking  is  focussing  on  temporal-spatial  uncertainty  of  contacts  in  a  Maritime  defence  and 
security  context,  there  are  a  number  of  levels  of  information  aggregation  in  developing  the 
Recognized  Maritime  Picture  (RMP),  and  each  level  of  aggregation  may  have  its  own 
uncertainty  visualization  requirements. 

The  techniques  of  uncertainty  visualization  reviewed  in  this  project  were  categorized  as  either 
intrinsic  (modification  of  the  primary  data  sign)  or  extrinsic  (addition  to  the  primary  data  sign.) 
Maritime  RMP  displays  often  use  four  intrinsic  attributes  of  icons,  position,  orientation,  shape 
and  colour,  to  indicate  primary  contact  data  nominal  location,  velocity  (by  means  of  a  vector 


15  After  Schneiderman,  B.  (2002)  Inventing  discovery  tools:  Combining  information  visualization  with  data 
mining.  Information  Visualization,  1(1),  pp.  5-12. 
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add-on  glyph)  and  identification  category,  with  colour  and  shape  used  redundantly.  While 
varying  these  attributes  to  denote  or  connote  uncertainty  may  be  feasible,  effectiveness  remains 
to  be  determined,  particularly  as  some  of  these  features  may  become  indistinct  as  the  display 
field  of  view  is  increased.  Further,  there  are  international  symbology  standards  that  may 
preclude  significant  modification  of  the  primary  data  icon.  This  suggests  that  secondary  data 
such  as  uncertainty  should  be  displayed  extrinsically,  or  if  intrinsic  uncertainty  indications  are 
used,  there  should  also  be  extrinsic  indicators  that  are  displayed  redundantly. 

Pang  and  colleagues  (1997)  acknowledged  that  altering  primary  data  icons  to  provide 
secondary  information  could  lead  to  inappropriate  salience.  They  suggest  that  designers  may 
want  to  consider  combined  techniques,  such  as  counterbalancing  icon  thickness  with  decreased 
brightness  to  maintain  a  level  of  salience  that  is  commensurate  with  the  importance  of  the 
information. 

Brown  cites  Ehlschlaeger  et  al.  (1997),  suggesting  that  providing  the  operator  with  the  ability  to 
control  the  display  of  secondary  icons  or  suppressing  animation  may  exploit  the  attention- 
grabbing  characteristics  without  unduly  annoying  the  operator.  Such  an  approach  may  be 
advantageous  when  trying  to  visualize  the  track  history  of  a  contact  that  is  based  on  intermittent 
observations.  Schaefer  et  al.  (2004)  also  suggest  that  if  uncertainty  is  going  to  be  displayed  with 
primary  data,  it  should  be  available  on  demand.  Cultural  and  task-demand  differences  may  lead 
to  contexts  where  the  display  of  uncertainty  is  counter-productive.  For  example,  if  air  traffic 
controllers  are  unused  to  thinking  about  uncertainty,  displaying  it  may  create  additional  workload 
and  confusion  during  time-sensitive  decision  making.  Schaefer  et  al.  suggest  that  uncertainty 
visualization  may  be  more  relevant  to  longer  term  planning  than  immediate  control  and  that 
displaying  uncertainty  may  well  lead  to  overly  complex  displays  -  less  may  be  more  and  it  is 
likely  context  dependent. 

Colour  hue  is  already  used  as  a  primary  data  intrinsic  indicator,  redundant  with  shape,  for 
contact  symbol  borders  on  tactical  displays.  This  suggests  that  neither  hue  nor  shape  would  be 
appropriate  for  indicating  uncertainty,  at  least  for  the  symbol  borders,  as  they  already  have  a 
role  conveying  information  and  alteration  may  lead  to  misunderstanding  of  the  primary 
information.  Using  symbol-fill  hue  variation  to  indicate  uncertainty  would  have  to  be  assessed 
for  salience  and  interference  with  the  primary  data  indicators.  Further,  although  colour  is 
fundamentally  an  ordered  quantity  due  to  its  dependency  on  wavelength,  it  is  unlikely  to  be 
perceived  that  way  automatically  suggesting  the  correspondence  between  hue  and  uncertainty 
would  have  to  be  learned,  creating  opportunities  for  failure. 

Symbol  brightness  or  transparency  may  lead  to  inappropriate  saliency  that  distracts  the  operator 
even  though  the  literature  suggests  they  may  be  effective  indicators  for  ordered  information 
such  as  levels  of  uncertainty.  If  these  intrinsic  characteristics  were  applied  under  operator 
control,  however,  they  may  be  effective  means  to  quickly  assess  which  contacts  require 
refreshed  data.  A  similar  approach  might  be  considered  for  applying  animation. 

The  extrinsic  techniques  seem  to  have  the  greatest  potential  for  unambiguous  representation  of 
uncertainty,  although  that  is  not  guaranteed.  The  extrinsic  techniques  do  have  the  potential 
disadvantage  of  adding  clutter  to  the  display,  so  it  is  likely  that  they  should  be  used  with 
operator  control  as  well.  As  threat  rings  are  used  as  secondary,  important  contact  information 
on  tactical  plots,  using  extrinsic  area  graphics  similar  to  the  sensor  area  coverage  use  in  Figure 
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2-7  to  indicate  location  uncertainty  may  be  inappropriate  unless  it  is  applied  under  operator 
control.  Other  extrinsic  methods  such  as  the  Lego  or  Rectangle  methods  of  Figure  2-6  may 
provide  a  compromise  between  clutter  and  intelligibility  if  it  is  found  that  continuous  display  of 
uncertainty  is  advantageous.  However,  there  should  be  clear,  culturally  semantic  associations 
between  the  meanings  of  such  glyphs  in  the  applied  context. 

Intrinsic  and,  to  a  lesser  extent,  extrinsic  approaches  to  representing  the  extent  of  temporal  or 
geospatial  uncertainty  may  be  somewhat  ambiguous  as  indicated  by  participants  confusing  the 
intended  associations  in  some  of  the  experimental  studies.  This  suggests  that  a  graphical 
display  strategy  for  depicting  uncertainty  of  primary  data  should  be  kept  simple  if  it  is  symbolic 
rather  than  displaying  a  physical  correspondence.  In  any  event,  the  user  interface  should  allow 
the  user  control  to  display  the  secondary  data  optionally  or  in  successive  levels  of  detail  to  delve 
deeper  to  expose  the  contributing  elements. 
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3  INDUSTRY  SURVEY 

The  industry  survey  was  undertaken  to  understand  what  is  currently  done  to  represent 
temporal-geospatial  uncertainty  in  data  fusion  graphics.  This  survey  begins  with  a  short  review 
of  ATC  tactical  displays  and  then  discusses  current  military  standards  that  are  shared  among 
western  nations  before  moving  onto  specific  examples  in  the  Royal  Canadian  Navy  (RCN).  The 
survey  then  considers  private  industry  organizations  who  are  actively  dealing  with  data  fusion 
and  uncertainty. 

A  traditional  keyword  search  involving  various  combinations  and  variations  of,  graphics,  data 
fusion,  geospatial,  uncertainty  and  company  failed  to  indicate  industry  representatives  who  are 
active  and  dedicated  to  the  development  of  uncertainty  visualizations  for  data  fusion  graphics. 
Instead,  industry  knowledge  was  used  to  enter  the  participant  and,  thereafter,  the  ‘path’  being 
uncovered  was  followed. 

3.1  Air  Traffic  Control 

Air  traffic  control  (ATC)  was  considered  because  it  uses  a  tactical  situation  display  (TSD)  that  is 
similar  to  military  tactical  plots,  although  the  industry  uses  a  different  approach  to  data  fusion 
than  seen  in  military  domains.  There  are  two  reasons  for  the  different  approach:  (a)  there  are 
fewer  sensors  being  used  and  (b)  the  airspace  is  generally  populated  by  cooperative  targets.  In 
ATC  only  radar  (primary  and  secondary)  are  being  used  to  plot  the  location  of  contacts.  Since 
ATC  is  generally  a  cooperative  practice,  all  aircraft  in  the  system  are  providing  identity 
information,  so  there  are  few  unknowns  and  therefore  less  need  for  data  fusion  or  associated 
symbology.  The  exception  is  an  aircraft  that  is  not  providing  identification  information  but  these 
are  readily  identified  by  primary  radar  and  tracked  as  unknown  contacts. 

At  the  lowest  level,  ATC  practices  low-level  information  fusion  (LLIF)  by  fusing  data  from 
primary  radar,  Mode  A,  Mode  C  and  Mode  3.  These  sensors  are  all  radar  based  and  are 
therefore  highly  accurate.  Typically  the  primary  radar  video  is  suppressed  and  replaced  by 
Mode  A/C  symbols  (in  practice  Mode  3/C  are  the  same).  The  resulting  symbol  is  intrinsic,  as 
shown  in  Figure  3-1 . 


Figure  3-1 :  ATC  Contact  Symbol 

The  raw  radar  return  is  processed  and  represented  as  a  square;  the  '+’  in  the  middle  of  the 
square  represents  Mode  A  and  the  ‘x’  in  the  middle  of  the  square  represents  Mode  C.  However, 
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more  modern  ATC  displays  have  foregone  this  sort  of  symbology  due  to  the  high  reliability  of 
radar  and  transponder  systems  and  now  instead  use  small  aircraft  symbols. 

In  creating  the  symbol,  sensors  that  are  used  are  co-located.  The  ATC  system  does  not  fuse 
data  from  sensors  in  different  locations,  instead  adopting  a  ‘radar  mosaic’  approach.  In  this 
approach  the  airspace  is  divided  into  ‘tiles’.  Analysis  is  performed  for  each  tile  and  the  best 
radar  source  for  that  area  is  selected.  The  system  automatically  switches  the  radar  source  as  an 
aircraft  tracks  from  one  tile  to  another. 

3.2  Military  Standards 

The  most  widespread  military  standard  for  graphics  is  MIL-STD-2525C  (DOD,  2008).  MIL-STD- 
2525C  describes  common  warfighting  symbology  that  has  been  developed  using  human  factors 
research  to  eliminate  conflicts  between  symbol  sets  and  allow  information  sharing  between 
different  military  domains.  MIL-STD-2525C  has  also  been  adopted  by  the  NATO  and  uses  a 
standard  format  for  the  following  categories  of  contacts: 

•  Pending 

•  Unknown 

•  Friend 

•  Neutral 

•  Hostile 

•  Assumed  friend 

•  Suspect 

Extrinsic  symbol  modifiers  have  been  defined  that  may  be  added  to  the  primary  object  icon. 
These  extrinsic  modifiers  conform  to  the  template  shown  in  Figure  3-2.  Modifiers  have  been 
developed  for  each  of  the  following  geospatial  or  application  domains: 

•  Unknown 

•  Space 

•  Air 

•  Units 

•  Equipment 

•  Installations 

•  Sea  surface 

•  Sea  subsurface 

•  Special  operations  forces 
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Figure  3-2:  General  Layout  of  Military  Symbology 
Image  reproduced  from  MIL-STD-2525C  (DOD,  2008) 

Each  letter  or  letter  combination  indicates  an  elaborating  ‘field’  of  associated  information.  Of 
these  letter  or  letter  combinations,  two  are  concerned  with  uncertainty.  J  is  an  evaluation  rating 
concerned  with  reliability  of  the  information  and  credibility  of  the  source;  an  extrinsic  graphic 
may  be  applied  around  the  symbol  indicating  uncertainty  various  object  attributes  as  shown  in 
Figure  3-3.  Further  to  this,  the  textual  label  ‘unknown’  may  be  applied  to  make  uncertainty 
explicit. 


Figure  3-3:  Extrinsic  modifiers  showing  techniques  for  displaying  attributes  and 
uncertainty  of  the  primary  object  icon  (blue  filled  circle). 

Image  reproduced  from  (DOD,  2008). 


09  June  2015  -  26  -  5769-001  Version  03 

©  Her  Majesty  the  Queen  in  Right  of  Canada,  as  represented  by  the  Minister  of  National  Defence,  2015 


CAE 


Visualizing  Uncertainty  for  Data  Fusion  Graphics 
Review  of  Selected  Literature  and  Industry  Approaches 


As  an  intrinsic  example  of  representing  information,  the  various  object  symbols  for  sea  surface 
track  contacts  are  provided  in  Figure  3-4.  Contacts  are  distinguished  both  by  shape  and  colour; 
the  text  labels  were  added  to  the  symbols  for  this  report  to  identify  the  icon  representation. 


Unknown  Friend  Neutral  Hostile 


Figure  3-4:  Sea  surface  contact  symbols  showing  extrinsic  icon  representations. 

The  text  labels  are  added  for  illustration  purposes  only.  Icon  images  reproduced  from 

MIL-STD-2525C  (DOD,  2008) 

The  examples  above,  while  making  explicit  reference  to  uncertainty,  they  do  so  qualitatively  and 
do  not  explicitly  express  the  magnitude  temporal  or  geospatial  uncertainty.  Indeed,  MIL-STD- 
2525C  make  no  mention  of  geospatial  uncertainty  and  leaves  the  specific  depiction  of  graphics 
associated  with  areas  of  uncertainty  and  the  calculations  associated  with  them  up  to  the 
national  authority  and  those  developing  military  tactical  displays. 

3.3  Royal  Canadian  Navy 

3.3.1  Data  Fusion 

The  RCN  has  a  modern  Command  and  Control  (C2)  system  that  includes  a  tactical  plot  that  is 
among  the  most  capable  of  allied  C2  systems.  Of  particular  interest  is  the  data  fusion  capability 
resident  in  the  Combat  Management  System  (CMS)  330  and  the  associated  graphics.  The  CMS 
330  includes  a  sophisticated  data  fusion  system  that  currently  combines  sensor  information  for 
the  purposes  of  unambiguously  locating  contacts  in  the  tactical  environment  and  providing  initial 
identification  based  on  kinematic  information.  Eventually  the  data  fusion  engine  will  incorporate 
intelligence  information  and  other  databases  to  amplify  the  identification  and,  ultimately,  infer 
the  intent  of  the  contact  through  pattern  matching  and  other  artificial  intelligence  methods. 

From  interviews  with  RCN  operators,  the  key  sensors  that  contribute  to  the  data  fusion  solution 
are  radar,  the  Automatic  Identification  System  (AIS)  and  tactical  data  links16  (i.e.  tracks  that  are 
provided  from  allied  ships  through  link  systems  such  as  LINK  16).  Other  sensor  information  is 
also  fused,  for  instance  electronic  warfare  bearings  and  acoustic  information.  The  data  fusion 
engine  integrates  geospatial  data  to  provide  an  instantaneous  contact  position  and  speed. 

Some  sensors  are  known  by  operators  to  be  more  or  less  reliable.  Accordingly,  operators  will 
adjust  the  priority  of  some  contributing  sensors  to  give  them  lower  priority  in  the  data  fusion 
solution  and,  in  some  situations,  may  remove  a  sensor  from  a  data  fusion  solution  entirely.  The 
relative  priority  of  sensors  in  the  data  fusion  solution  is  incorporated  into  a  Dempster-Shafer 
theory-based  calculation  that  assesses  the  degree  of  belief  in  the  information  from  one  sensor. 
The  fusion  algorithm  then  combines  that  degree  of  belief  with  those  for  all  other  sensors  before 
calculating  the  geospatial  position  of  the  contact. 


16  http://en.wikipedia.org/wiki/Tactical  Data  Link 
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Nevertheless,  there  is  no  representation  of  uncertainty  in  the  resulting  contact  location.  A 
contact  symbol,  similar  to  that  shown  in  Figure  3-4,  is  provided  at  the  location  of  the  contact  as 
determined  by  the  system.  Uncertainty  is  only  indicated  to  the  operator  when  different  sensor 
information  is  not  automatically  fused.  In  this  situation,  two  or  more  nearly-coincident  contact 
symbols  will  be  displayed.  In  this  case  it  is  up  to  the  operator  to  manually  decide  the  most 
reliable  contact  track  and  then  merge  other  symbols  with  it,  a  process  referred  to  as  ‘correlating 
contacts’.  In  this  case,  there  is  no  averaging  of  the  geospatial  information. 

3.3.2  Tactical  Graphics 

Modern  tactical  sensors  used  in  surface  and  air  warfare  domains  provide  reliable  and  accurate 
information,  such  as  position  or  velocity,  for  sensed  objects.  However,  there  can  be  a  great  deal 
of  uncertainty  in  the  underwater  domain  due  to  the  way  an  acoustic  signal  (based  on  pressure) 
interacts  with  the  water  and  sea  bed  features.  For  this  reason  there  have  been  a  number  of 
graphics  developed  for  the  Anti-Submarine  Warfare  (ASW)  team  to  help  them  manage  the 
tactical  picture.  The  following  Tactical  Decision  Aides  (TDA)  were  developed  for  the  CMS330 
and  the  images  are  reproduced  from  a  report  prepared  during  work  on  modernization  of 
Canada’s  Halifax-class  frigate  (CAE,  2011): 

•  Water  Space  Management:  this  tool  deconflicts  underwater  operations  where  there  is 
inherent  uncertainty  in  the  location  of  the  subsurface  assets. 

•  Torpedo  Threat  Reckon:  this  tool  displays  a  graphical  object  based  on  a  sensor  ‘glimpse’  of 
a  torpedo  that  proceeds  toward  the  ownship  on  a  specified  bearing  at  a  specified  speed; 
note  that  a  similar  TDA  is  available  for  missiles  that  may  be  glimpsed  before  travelling  in 
ways  to  evade  detection  -  see  Figure  3-5. 


Figure  3-5:  Torpedo  Threat  Reckon 
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•  Acoustic  Cross  Fix:  this  tool  will  plot  a  location  of  the  source  along  with  a  shaded  area 
defined  by  the  calculated  margin  for  error  on  the  sensor  based  on  two  or  more  acoustic 
bearings  -  see  Figure  3-6). 


Figure  3-6:  Acoustic  Cross  Fix 

•  Target  Motion  Analysis:  this  tool  uses  successive  acoustic  sensor  readings  to  develop  an 
accurate  location,  course  and  speed  for  a  submarine  -  see  Figure  3-7. 


Figure  3-7:  Target  Motion  Analysis  Display 

•  Limiting  Lines  of  Submerged  Approach  (estimates  the  possible  firing  position  of  a 
submarine) 
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•  Furthest  On  Circle  (estimate  the  distance  a  surface  or  submerged  target  has  travelled  since 
its  last  detection) 

•  Cone  Of  Courses  (indicate  the  most  likely  area  to  search  for  an  approaching  submarine, 
given  the  last  known  position  of  the  submarine) 


NOTE  that  these  last  three  TDAs  result  in  a  graphical  object  and  involve  the  fusion  of  different  sensor 

information  with  some  inputs  from  the  operator  to  generate  that  graphical  object.  The  uncertainty  is 
depicted  by  an  area  around  the  contact  (e.g.  similar  to  Figure  3-6);  although  this  does  not  apply  to 
the  Torpedo  Threat  Reckon  or  the  Target  Motion  Analysis  output. 


3.4  Private  Industry 

A  search  of  the  internet  uncovered  few  companies  that  are  specifically  in  the  business  of 
developing  data  fusion  graphics.  Rather,  companies  who  develop  graphical  applications  claimed 
to  incorporate  data  fusion  engines.  This  made  it  very  difficult  to  identify  companies  of  direct  and 
clear  relevance  to  this  work.  No  companies  made  specific  claims  concerning  the  development  of 
graphical  methods  for  representing  uncertainty  in  data  fusion  graphics.  Where  research  and 
examples  were  found,  however,  was  in  the  area  of  meteorological  prediction  of  storm  tracks  and 
similar  applications.  Their  level  of  precision  is  similar  to  the  ‘Furthest  On  Circle’  and  ‘Cone  Of 
Courses’  TDAs  described  above. 

It  is  known  from  other  work  that  military  combat  systems  integrators  develop  tailored  graphics 
solutions  for  tactical  displays.  For  this  reason,  the  largest  systems  integrators:  Lockheed  Martin, 
Thales,  British  Aerospace  (BAe),  SAAB,  General  Dynamics,  and  Rheinmetall,  were  all  used  in 
keyword  searches  with  the  terms  ‘uncertainty’  and  ‘graphic’  in  the  hope  that  trade  papers  would 
be  found.  Unfortunately  almost  all  returns  concerned  business  or  climate  applications.  When  the 
organization  names  were  searched  with  ‘data’  and  ‘fusion’  more  relevant  results  were  returned, 
but  these  dealt  with  algorithms  and  processing  required  to  achieve  data  fusion  rather  than  how 
to  graphically  present  the  results. 

3.4.1  OODA  Technologies 

OODA  Technologies17  was  considered  for  their  expertise  in  representation  of  uncertainty  in  data 
fusion  graphics.  OODA  is  located  in  Montreal  and  provides  services  for,  amongst  other  things, 
visualization  to  help  organizations  understand  and  navigate  data.  Their  website  includes  a 
number  of  project  examples,  including  several  with  potential  relevance  for  uncertainty 
visualization  in  data  fusion  graphics. 

•  RISOMDA  is  concerned  with  the  development  of  Maritime  Situation  Awareness  (MSA) 
products  to  support  wide  area  surveillance  for  ship  approaches  to  Canada  through  the 
Pacific,  Arctic  and  Atlantic  oceans.  This  work  includes  integrating  a  variety  of  sensor  data 
and  has  included  studies  of  user  trust  in  the  data  model,  a  key  component  of  data  fusion 
and  once  closely  related  to  uncertainty. 


17  https://ooda-staqinq.herokuapp.com;  www.ooda.ca 
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•  BullBear  is  a  commodity  trading  application  for  financial  purposes.  In  common  with 
geospatial  data,  the  BullBear  application  incorporates  information  from  a  variety  of  sources 
(generally  text-based  news  articles)  and  then  assigns  a  score  to  the  commodity.  The  score 
(BullBear  Index)  is  the  average  of  all  other  indexes  computed  from  all  sentences  in  the  last 
20  relevant  news  articles  for  the  previous  two  weeks.  This  averaging  is  conceptually  similar 
to  the  data  fusion  approach  in  the  CMS330. 

•  PASE  (Persistent  Active  Surveillance  of  the  Exclusive  Economic  Zone)  fuses  radar  with  AIS 
information  based  on  track  kinematics  and  geospatial  information.  There  is  no  mention  of 
whether  uncertainty  is  dealt  with  although  it  is  expected  that  an  averaging  approach  to 
location  is  used  and  no  uncertainty  is  represented  to  operators. 

OODA  Technologies  actively  publish  their  work  in  a  variety  of  media. 

3.4.2  Menya  Solutions 

Menya  Solutions18  was  considered  for  their  expertise  concerning  data  fusion  and  visualization  of 
uncertainty.  Menya  Solutions  is  based  in  Sherbrooke,  QC  and  focuses  on  automating  and 
supporting  real-time  decision  making  in  complex  environments.  Their  solutions  fall  into  two  main 
categories: 

•  Automated  behaviour  recognition,  machine  learning  and  complex  event  processing. 

•  Simulation  and  planning.  This  involves  the  modelling  of  a  work  domain  and  simulating 
scenarios  stochastically  to  generate  a  repertoire  of  case  based  reasoning  exemplars. 

The  Menya  Solutions’  website  includes  several  project  examples  with  potential  relevance  for 
uncertainty  visualization  in  data  fusion  graphics: 

•  ICOR  is  an  application  for  understanding  events  as  they  occur  in  real-time.  Sensor  data  is 
collected  and  combined  with  spatial  and  temporal  relationships  among  contacts  to  permit 
pattern  recognition,  goal  (of  the  contact)  recognition,  plan  recognition  and  threat  recognition. 
There  is  no  description  of  how  this  information  is  presented  to  the  operator,  if  at  all. 

•  SimPlan  is  a  decision  support  and  optimizer  tool  for  simulations. 

Menya  Solutions  recently  won  a  contract  with  DRDC  Valcartier  to  work  on  the  Naval  Battle 
Management  Command  and  Control  Decision  and  Collaboration  Support  project  with  the 
specific  objective  of  developing  the  RCN  capability  in  Integrated  Air  and  Missile  Defence 
(IAMD).  A  current  variant  of  what  will  eventually  be  delivered  from  this  work  is  in  service  as  an 
integral  part  of  the  CMS330. 


18  http://menvasolutions.com/ 
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3.4.3  Uncharted 

Uncharted19  (formerly  Oculus  Info  Inc.)  were  suggested  by  DRDC  scientists.  Uncharted 
develops  tools  for  analyzing  data.  Uncharted  has  both  planning  and  real-time  applications  that 
have  been  applied  to  defence  and  intelligence  applications.  Uncharted’s  product  GeoTime 
represents  events  spatially  and  temporally,  allowing  the  analyst  to  animate  the  visualization  and 
see  how  events  have  unfolded  over  time.  This  allows  patterns  and  trends  to  be  identified  and 
studied. 

Although  most  of  the  applications  listed  by  Uncharted  seem  to  be  retrospective  analysis, 
Uncharted  does  list  ‘GPS  and  Location-Based  Services  (Emergency,  Asset  Tracking)’  as  one 
application,  which  could  be  considered  analogous  to  real-time  sensor  tracking.  A  .NET 
Application  Program  Interface  (API)  exists  that  may  allow  GeoTime  to  incorporate  real-time 
sensor  data  (e.g.  radar,  AIS).  GeoTime  is  also  compatible  with  ArcGIS,  which  further  enhances 
the  ability  to  fuse  a  tactical  picture  with  underlying  geographical  chart  information. 

While  GeoTime  has  not  been  used  for  real-time  sensor  data  fusion,  it  has  been  used  for 
intelligence  data  fusion  on  the  basis  of  pattern  search  and  temporal  link  analysis.  However,  their 
website  does  not  make  any  reference  to  uncertainty  in  their  representations. 

Uncharted  also  have  a  long  list  of  publications  with  some  having  relevance  to  the  visualization 
of  uncertainty  for  data  fusion  graphics: 

•  Wright,  W.  &  Dupuis,  C.  (1997).  Resolving  Radar  Ambiguities:  A  3D  look  at  EW  Analysis. 
Journal  of  Electronic  Defence,  Sept. 

•  Wright,  W.  &  Kapler,  T.  (2002).  Visualization  of  Blue  Forces  Using  Blobology.  In 
International  Command  and  Control  Research  and  Technology  Symposium  (ICCRTS), 

June. 

•  Kapler,  T.  &  Wright,  W.  (2002).  Visualization  of  Self-Reporting  Contacts  for  Command  and 
Control.  In  International  Command  and  Control  Research  and  Technology  Symposium 
(ICCRTS),  June. 


19  http://uncharted.software/ 
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4  GENERAL  DISCUSSION 

Almost  20  years  ago,  Andre  and  Cutler  (1998)  observed  that  it  was  a  generally  held  view  that  it 
is  important  to  consider  uncertainty  in  aviation  decision  making  but  that  little  research  had  been 
done  on  how  best  to  incorporate  uncertainty  in  displays  of  information  relevant  to  navigation 
tasks.  That  insufficient  foundational  research  has  been  done  was  restated  by  Riveiro  (2007) 
almost  ten  years  later  and  while  some  progress  has  been  made  since  Andre  and  Cutler’s  time, 
the  issue  is  still  appears  to  be  unresolved  (Bisantz,  2013).  What  is  interesting  is  that  few 
question  Andre  and  Cutler’s  premise  that  visualizing  uncertainty  is  important  (see  Harrower, 
2003,  for  a  discussion  of  this  point.)  It  is  interesting  that  so  few  of  the  systems  in  use  or  their 
manufacturers  describe  capabilities  for  displaying  data  uncertainty. 

In  the  papers  reviewed  for  this  tasking,  the  need  for  visualization  was  often  stated  but  often  only 
justified  in  rather  vague  terms.  Perhaps  the  authors  possessed  an  implicit  understanding  of  the 
underlying  goals  and  tasks,  but  the  tasks  and  associated  goals  for  operators  in  the  Maritime 
defence  and  security,  as  described  in  a  variety  of  Hierarchical  Task  Analyses,  did  not  seem  to 
explicitly  document  the  need  to  represent  uncertainty.  Nor  was  a  Cognitive  Work  Analysis 
reviewed  that  delved  into  the  constraints  and  decisions  these  operators  faced.  Some  insight  into 
this  problem  space  may  be  available  reports  by  Matthews  and  colleagues  (Matthews,  Webb,  & 
Bryant,  1999;  Matthews,  Webb,  &  Keeble,  2002;  Matthews,  Webb,  &  Woods,  2001),  however, 
there  was  insufficient  time  to  review  these  documents.  Nevertheless,  discussions  with  CAE 
personnel  who  had  relevant  experience  was  CAF  operators  in  the  Maritime  defence  and 
security  domain  acknowledge  that  uncertainty  information  was  useful  during  tactical  decision 
making. 

Avram  (2012)  notes  that  a  great  deal  of  sensor  fusion  visualization  research  is  on  anomaly 
detection,  which  is  generally  considered  part  of  high  level  information  fusion  (HLIF)  involving 
temporal-geospatial  data  as  well  as  meta-data.  Anomaly  detection  in  the  Maritime  defence  and 
security  domain  typically  focusses  on  trying  to  identify  vessels  that  are  behaving  in  an  unusual 
manner  compared  with  lawful  vessels  operating  in  the  area  -  an  applied  case  of  target  detection 
among  distractors.  Anomaly  detection  is  a  complex  problem,  often  involving  many  vessels  that 
may  be  observed  by  different  sensors  in  a  distributed  network  leading  to  different  levels  of 
accuracy  or  timing.  Further,  there  are  many  situations  where  behaviour  may  be  anomalous  yet 
lawful  (ill-defined  problem  space.)  However,  HLIF  tasks  typically  require  understanding  of 
temporal-spatial  information  resulting  from  LLIF  tasks,  but  exactly  how  these  data  are  used  by 
domain  SMEs  is  unclear  from  the  papers  reviewed  here. 
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5  RECOMMENDATIONS 

The  following  recommendations  are  intended  to  help  clarify  how  depiction  of  temporal  or 

geospatial  uncertainty  can  be  effectively  displayed  to  Maritime  defence  and  security  operators. 

Some  of  the  information  may  already  exist  in  documents  that  were  not  reviewed  or  even 

identified. 

1 .  Identify  or  establish  an  explicit  description  of  the  goals  and  activities  of  Maritime  defence 
and  security  decision  makers  at  various  levels  of  command  that  require  temporal-geospatial 
information.  This  should  include  a  description  of  the  information  requirements,  in  particular 
the  sources  of  uncertainty,  the  nature  of  the  uncertainty  (i.e.  whether  it  is  intransient 
knowledge  or  moment-by-moment,  case-by-case.)  This  should  also  establish  unequivocally 
that  operators  can  use  uncertainty  information  effectively  in  their  reasoning  and  decision 
making  independent  of  the  method  of  providing  the  uncertainty  information. 

2.  Identify  or  establish  through  a  CWA  the  constraints  imposed  on  the  display  of  sensor  fusion 
data  that  might  affect  an  operator’s  ability  to  use  the  information  if  the  data  is  available  to  be 
displayed.  This  should  identify  the  Human  Factors  constraints  that  affect  visualizing 
temporal  or  spatial  data  based  on  psychologically  established  perceptual  and  reasoning 
capabilities.  This  work  could  consider  multi-modal  information  displays  that  are  consistent 
with  the  constraints  imposed  by  the  operational  domain. 

3.  Select  or  develop  an  extrinsic  or  a  combined  extrinsic  and  intrinsic  approach  using 
techniques  that  have  been  shown  to  be  effective  for  depicting  secondary  information  such 
as  uncertainty,  then  experimentally  validate  their  effectiveness  for  improving  performance  in 
an  application  context  that  suggests  comprehension  of  primary  data  uncertainty  from  sensor 
fusion  is  advantageous.  This  should  also  incorporate  a  measure  of  operator  control  over  the 
display  of  uncertainty  information  and  how  best  to  implement  such  control  in  an  operational 
setting. 
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6  CONCLUSION 

This  review  of  the  scientific  literature  and  industry  state  of  practice  suggests  that  work  remains 
to  be  done  to  establish  how  uncertainty  should  be  indicated  on  Maritime  defence  and  security 
graphical  displays  of  fused  sensor  data.  Although  some  fundamental  information  of  how 
operators  perceive  displayed  information  exists,  refinement  of  that  understanding  should  be 
undertaken  to  establish  which  methods  are  feasible  and  effective  from  a  Human  Factors 
perspective. 

Also,  the  need  for  visualization  of  uncertainty  in  defence  information  displays  needs  to  be 
established  based  on  CWA  assessment  of  the  domain  constraints  and  on  HTA  assessment  of 
the  operators  goals  or  objectives  that  are  supported  by  scientifically  sound  operational 
effectiveness  analyses. 

In  the  interim,  a  simple,  extrinsic  indication,  or  a  combined  extrinsic  and  intrinsic  indication,  of 
primary  data  uncertainty  that  is  under  the  operator’s  control  would  seem  to  be  the  most 
promising  approach  to  ensure  comprehension  while  limiting  clutter. 
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8  DEFINITIONS 


The  following  definitions  were  adopted  by  the  authors  to  have  a  shared  understanding  of  this 
call-up’s  objectives.  The  definitions  are  derived  from  those  found  in  the  public  domain  either  as 
formal  definitions,  such  as  the  U.S.  Department  of  Defense  Data  Fusion  Subgroup,  or  as 
informal  refinements  of  colloquial  expressions  used  in  reports  by  other  authors  in  published 
reports. 

8.1  Contact 

A  contact  is  a  physical  object  or  a  sensor  indication  of  an  object  that  may  be  of  interest  to  an 
operator.  In  a  Maritime  domain,  contacts  typically  comprise  ships,  boats,  submarines,  airplanes 
or  helicopters,  but  may  also  include  wildlife  or  erroneous  sensor  indications  of  a  physical  object. 


8.2  Colour20 


Colour  is  a  property  of  an  object  that  is  perceived  visually  due  to  emitted  or  reflected  light.  There 
are  numerous  factors  that  can  affect  colour,  but  it  is  chiefly  determined  by  hue,  saturation  and 
lightness. 

8.3  Data  and  Information 

Data  and  information  are  subtly  different  concepts  with  information  usually  implying  some  form 
of  processing  or  organization  of  data.  This  distinction  seems  unnecessary  in  the  current  context 
and  the  two  words  will  be  used  interchangeably. 

8.4  Geospatial 

Geospatial  refers  to  the  geographical  coordinates  (latitude,  longitude,  altitude,  depth)  used  to 
locate  a  contact  in  the  AOR. 

8.5  High  Level  Information  Fusion 

High  level  information  fusion  (HLIF)  is  generally  considered  to  be  the  process  of  consideration 
and  integration  of  abstract  information  that  is  important  to  understanding  a  tactical  situation, 
inferring  a  threat  from  contact  behaviour  or  predicting  implications  for  the  tactical  situation 
(Blasch,  Valin,  Jousselme,  Lambert,  &  Bosse,  2012). 


8.6  Hue 


Hue  is  an  attribute  of  perceived  light  that  is  commonly  referred  to  as  colour  and  often 
categorized  by  the  stimulus  similarity  to  pure  red,  orange,  yellow,  green,  blue,  indigo  or  violet. 


20  Definitions  related  to  colour  attributes  were  paraphrased  from  http://www.workwithcolor.com/color-properties- 
definitions-0101.htm  and  Wikipedia. 
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8.7  Lightness 

Lightness  is  the  degree  to  which  a  colour  hue  is  modified  by  adding  a  quantity  of  either  black  or 
white.  Lightness  is  commonly  referred  to  as  tints  (when  white  is  added)  or  shades  (when  black 
is  added.)  Lightness  is  sometimes  confused  with  brightness,  which  is  a  characteristic  the 
amount  of  light  being  emitted  or  reflected  by  an  object  (i.e.  luminance.) 

8.8  Low  Level  Information  Fusion 

Low  level  information  fusion  is  generally  is  generally  considered  to  be  either  the  process  of 
integration  of  raw  electronic  or  digital  sensor  signals  into  human-interpretable  data  or  the 
consideration  and  integration  of  such  interpretable  data  into  information  that  supports  target 
classification,  identification  or  tracking  (Blasch  et  al.,  2012). 

8.9  Operator 

For  the  purposes  of  this  report,  an  operator  is  considered  to  be  a  person  who  uses  a  visual 
display  of  information  to  manipulate  the  data,  reason  about  the  data  or  make  decisions  based 
on  the  displayed  information.  In  the  tactical  Maritime  context,  operators  may  be  any  of  the 
following: 

•  TACCO  -  Tactical  coordination  officer  (Maritime  aviation:  Sea  King,  Cyclone); 

•  TACNAV  -  Tactical  navigator  (Maritime  aviation:  Aurora); 

•  NESOP  -  Naval  Electronic  Support  Operator  (Navy:  HALIFAX-Class  Frigate); 

•  NCIOP  -  Naval  Combat  Information  Operator  (Navy:  HALIFAX-Class  Frigate); 

•  SWC  -  Surface  Weapons  Controller  (Navy:  HALIFAX-Class  Frigate); 

•  ASWC  -  Anti-submarine  Weapons  Controller  (Navy:  HALIFAX-Class  Frigate);  and, 

•  ORO  -  Operations  Room  Officer  (Navy:  HALIFAX-Class  Frigate). 

8.10  Recognized  Maritime  Picture 

The  RMP  is  the  presentation  of  geo-located  contacts  on  an  electronic  map  display  intended  to 
allow  management  of  information  derived  from  an  array  of  sources  (including  sensors)  and 
support  tactical  decision  making  (Lapinski,  2010). 

8.11  Saturation 

Saturation  is  a  property  of  perceived  light  that  indicates  the  level  of  pure  colour  (100% 
saturation)  in  a  mixture  with  pure  gray  (0%  saturation). 


09  June  2015 


-41  - 


5769-001  Version  03 


©  Her  Majesty  the  Queen  in  Right  of  Canada,  as  represented  by  the  Minister  of  National  Defence,  2015 


CAE 


Visualizing  Uncertainty  for  Data  Fusion  Graphics 
Review  of  Selected  Literature  and  Industry  Approaches 


8.12  Sensor 

A  sensor  is  a  physical  device  that  is  capable  of  measuring  some  physical  property  of  an  object 
of  interest.  Sensors  that  are  relevant  to  the  current  review  for  the  Maritime  domain  include: 

•  Acoustic  (sonobuoy); 

•  Automatic  Identification  System  (AIS); 

•  Electro-optical; 

•  Electronic  Support  Measures  (ESM); 

•  Infra-red  (optical); 

•  Long  range  identification  and  tracking  (LRIT); 

•  RADAR  (imaging,  fire  control,  long  range,  short  range,  weather); 

•  Satellite  imagery;  and, 

•  Traffic  collision  avoidance  system  (TCAS). 

8.13  Sensor  fusion 

Sensor  fusion  has  been  defined  a  number  of  ways21,  no  doubt  varying  depending  upon  the 
context  in  which  the  author  was  applying  the  term.  The  U.S.  Department  of  Defense,  Joint 
Directors  of  Laboratories  (JDL)  created  a  formal  definition  in  the  mid-1980s  that  is  frequently 
referred  to,  but  extensions  have  been  proposed  over  the  intervening  years  (Foo  &  Ng,  2013; 
David  L  Hall  et  al.,  2001;  Steinberg  &  Bowman,  2001, 2004). 

Fusion  Levels  2  through  5  of  the  6  levels  of  the  JDL  Data  Fusion  Information  Group22  (DFIG) 
model  support  higher  level  reasoning  and  decision  making.  Although  these  information  fusion 
levels  are  generally  involved  with  more  abstract  information,  they  have  their  own  associated 
uncertainty  and  visualization  requirements.  Conversely,  Fusion  Level  0  -  Source  or  signal 
preprocessing  -  is  involved  with  the  processing  of  the  raw  electronic  or  digital  sensor  signals 
that  produce  the  information  presented  to  the  operator.  This  information  is  usually  presented 
graphically  (e.g.  an  acoustical  gram),  symbolically  on  a  tactical  plot,  or  as  alpha-numeric 
information  that  accompanies  the  graphical  plot. 

The  current  work  is  focussed  on  Level  1  -  Object  Assessment  -  although  there  is  some  overlap 
with  both  Level  0  (Source  preprocessing)  and  Level  2  (Situation  Assessment).  Level  1  fusion  is 
concerned  with  correlating  the  contact  level  indicators  or  signs  to  merge  redundant  symbols, 
localize  the  resultant  object  (geographical  coordinates,  altitude/depth,  speed,  heading,  range, 


21  http://net.cs.uni-bonn.de/wq/sensor-data-and-information-fusion/what-is-it/sdf-definitions/ 

22  http://en.wikipedia.org/wiki/Data  fusion 
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bearing,  time  of  observation)  and  identify  it  uniquely  so  that  the  contact  can  be  further 
processed  or  reasoned  about  in  subsequent  situation  assessment  activities23.  Results  of  the 
Level  1  fusion  process  are  increasingly  shared  among  networked  operators  who  are  attempting 
to  construct  a  common  understanding  of  the  tactical  environment  for  an  entire  task  force. 

8.14  Tactical  situation  display 

The  tactical  situation  display  (TSD),  also  referred  to  colloquially  as  the  tacplot  or  tactical  plot,  is 
a  computer  generated  map  showing  the  locations  and  associated  textual  information  for  contact 
contacts  in  an  area  of  interest. 

8.15  Time-late 

Time-late  is  an  expression  used  to  indicate  the  recency  of  information.  It  is  used  to  refer  to  the 
elapsed  time  between  the  observation  and  the  time  the  information  is  being  considered,  usually 
the  current  time. 

8.16  Uncertainty 

Uncertainty  may  comprise  a  number  of  different  concepts  including  error,  ambiguity, 
incompleteness  or  deception.  Uncertainty  indicates  a  lack  of  reliability  of  a  nominal  value.  While 
uncertainty  has  implications  for  the  concept  of  risk,  uncertainty  is  distinct  from  risk,  most  notably 
as  it  does  not  entail  a  concept  of  loss,  injury  or  adverse  consequences.  Uncertainty  may  be 
represented  as  a  numerical  probability,  but  it  can  be  indicated  graphically  as  a  categorical 
representation,  such  as  a  probable  location  area  versus  an  improbable  location  area  for  a 
contact  on  a  tactical  map  plot. 

Within  the  context  of  this  document,  uncertainty  is  considered  to  be  the  counterpart  of  accuracy 
and  reflects  the  possible  variability  of  a  datum  about  its  nominal  value.  More  specifically,  this 
report  is  concerned  with  the  variability  associated  with  the  nominal  values  of  a  contact’s 
attributes  (i.e.  latitude,  longitude,  et  cetera)  and  recency  of  the  observation. 

Zuk  (2008)  presents  an  extensive  typology  for  describing  uncertainty  that  may  be  used  to 
identify  contributing  factors  associated  with  the  reliability  of  information.  But,  while  it  may  be 
important  to  be  able  to  deconstructed  fused  information  through  the  use  of  data  fission  tools,  it 
is  not  clear  that  it  is  necessary  to  visualize  separate  contributions  of  uncertainty  to  denote  the 
reliability  of  the  primary  data  sign  or  icon. 

8.17  Visualization 

The  Oxford  dictionary24  defines  the  verb  “visualize”  to  be  1 .  To  form  a  mental  image  (of 
something);  to  imagine  (something);  and  2.  To  make  (something)  visible  to  the  eye.  The  current 
review  involves  both  of  these  definitions  as  it  is  focussed  on  visual  display  techniques  intended 


23  http://defensesvstems.com/articles/2009/09/02/c4isr1-sensor-fusion.aspx 

24  http://www.oxforddictionaries.com/definition/enqlish/visualize 
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to  support  the  operator’s  understanding  of  the  location  of  contacts  within  the  AOR  over  time 
(past,  present  and  future.) 
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9  ACRONYMS 


AIS 

AOR 

API 

ASW 

Automatic  Identification  System 

Area  of  operations 

Application  Program  Interface 

Anti-Submarine  Warfare 

C2 

CAF 

CMRE 

CMS 

CTFG 

Command  and  Control 

Canadian  Armed  Forces 

Centre  for  Maritime  Research  and  Experimentation 

Combat  Management  System 

Canadian  Tracking  and  Fusion  Group 

DND 

DRDC 

DTIC 

Department  of  National  Defence 

Defence  Research  and  Development  Canada 

Defense  Technical  Information  Center 

ETURWG 

Uncertainty  Reasoning  Working  Group 

FACTS 

Future  Area  Control  Tools  Support 

GPS 

Global  Positioning  System 

HFM 

HLIF 

HTA 

Human  Factors  and  Medicine 

High  Level  Information  Fusion 

Hierarchical  Task  Analysis 

IEEE 

InfoVis 

ISIF 

Institute  of  Electrical  and  Electronics  Engineers 

Information  Visualization 

International  Society  of  Information  Fusion 

LLIF 

Low  Level  Information  Fusion 

NATO 

NASA  TLX 

North  Atlantic  Treaty  Organization 

National  Aeronautical  and  Space  Administration  Task  Load  Index 

PDF 

PPD 

Portable  Document  Format  (Adobe®) 

Potential  Problem  Display 

RCN 

RMP 

RTO 

Royal  Canadian  Navy 

Recognized  Maritime  Picture 

Research  and  Technology  Organization 

SAR 

SAS 

SciVis 

SET 

STO 

Search  and  Rescue 

System  Analysis  and  Studies 

Scientific  Visualization 

Sensors  and  Electronics  Technology 

Science  and  Technology  Organization 
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Tacplot 

TDA 

TSD 

VAW 
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Tactical  plot 

Tactical  Decision  Aides 

Tactical  Situation  Display 

Vertical  Assistance  Window 


-46- 


5769-001  Version  03 


©  Her  Majesty  the  Queen  in  Right  of  Canada,  as  represented  by  the  Minister  of  National  Defence,  2015 


CAE 


Visualizing  Uncertainty  for  Data  Fusion  Graphics 
Review  of  Selected  Literature  and  Industry  Approaches 


APPENDIX  A  ANOTATED  BIBLIOGRAPHY  OF  RELATED 

DOCUMENTS 

The  following  documents  were  reviewed  and  while  they  did  not  have  information  that  was 
directly  applicable  to  the  central  issue  of  uncertainty  visualization  of  geospatial  data,  they  did 
contain  associated  information  that  the  client  expressed  some  interest  in.  For  that  reason,  the 
documents  are  included  in  this  appendix  with  notes  taken  by  the  reviewers. 

Atrey,  P.K.,  &  Hossain,  M.A.  (2010)  Multimodal  fusion  for  multimedia  analysis:  a  survey. 
Multimedia  Systems,  16:345-379 

This  paper  is  intended  to  provide  multimedia  researchers  with  a  state-of-the-art  of  fusion 
strategies  that  are  used  for  combining  multiple  modalities  in  order  to  accomplish  various 
multimedia  analysis  tasks.  The  authors  classify  the  multimodal  fusion  research  literature  by  the 
fusion  methodology  and  the  level  of  fusion.  For  each  method,  the  paper  describes  the  basic 
concept,  identifies  advantages  and  weakness,  and  notes  some  applications. 

The  authors  categorized  multimodal  fusion  approaches  as  one  of  three  types  with  subdivisions: 

1 .  Rule-based  fusion  methods 

a.  Linear  weighted  fusion  (widely  used) 

b.  Majority  voting 

c.  Custom-defined  rules  (flexible) 

2.  Classification-based  fusion  methods 

a.  Support  vector  machine 

b.  Bayesian  inference  (easy  to  extend  but  brittle) 

c.  Dempster-Shafer  theory 

d.  Dynamic  Bayesian  networks  (DBN;  applied  to  time-series  data) 

e.  Neural  networks 

f.  Maximum  entropy  model 

3.  Estimation-based  fusion  methods  (localization  and  tracking  tasks) 

a.  Kalman  filter 

b.  Particle  filter 
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Zuk,  T.  (2008).  Uncertainty  Visualization.  Presentation  at  the  InfoVis  -  Winter  2008. 

This  is  a  Power  Point  Presentation  introducing  a  group  of  methods  used  for  uncertainty 
visualization  although  data  fusion  is  not  the  focus  in  this  presentation.  This  presentation 
provides  an  uncertainty  typology,  including  accuracy/error,  precision,  completeness, 
consistency,  lineage,  currency/timing,  credibility  and  interrelatedness. 

The  presenter  identified  four  types  of  uncertainty  visualization  approaches  that  consist  of 
taxonomic  approach,  semiotic,  perceptual  and  cognitive  theory,  as  well  as  human  factors  and 
task  constraints.  For  the  taxonomic  approach,  the  presenter  defined  four  types  of 
attributes/data,  including  datum  values,  location  of  the  datum,  visualization  axes  mapping  and 
extend  of  both  location  and  values. 

The  author  of  the  presentation  identified  seven  methods  of  encoding  for  uncertainty 
visualization:  (1)  add  glyphs,  (2),  add  geometry,  (3)  modify  geometry,  (4)  modify  attributes,  (5) 
animation,  (6)  sonification,  and  (7)  psycho-visual.  For  the  semiotic  approach,  the  author 
introduced  five  representation  methods,  e.g.  crispness,  resolution,  transparency,  virtual 
reconstructions  and  molecular  positional  uncertainty. 

Some  key  points  relevant  to  human  factors  of  visualization  and  decision  making  were  showed  in 
the  slides,  including  heuristic  evaluation,  local  contrast  affects,  colour  and  gray  perception.  The 
author  also  introduced  the  Bayesian  theorem  for  evidence-based  diagnosis.  The  author  listed 
some  future  directions  for  uncertainty  visualization,  including  providing  support  for  cognitive  task 
simplification,  supporting  emphasis  and  de-emphasis  of  uncertainty  information,  supporting 
viewing  of  uncertainty  as  metadata  and  as  separate  data,  allowing  the  user  to  select  realizations 
of  interest,  mitigating  cognitive  heuristics  and  biases  with  reasoning  support,  providing 
interaction  to  assist  knowledge  creation,  and  assessing  the  implications  of  incorrectly 
interpreting  the  uncertainty. 

The  presenter  showed  6  images  for  the  representation  for  uncertainty:  (a)  vector  fields,  (b) 
temporal  uncertainty  in  Archaeology,  (c)  crispness,  (d)  resolution,  (e)  transparency,  (f)  virtual 
reconstructions,  and  (g)  molecular  positional  uncertainty. 

Blasch,  E.P.,  Dorion,  E.,  Valin,  P.,  Bosse,  E.,  and  Roy  Jean.  (2010).  Ontology  Alignment  in 
Geographical  Hard-Soft  Information  Fusion  Systems.  Proceedings  of  the  13th 
Conference  on  Information  Fusion  (FUSION  2010). 

This  paper  explores  issues  of  fusing  hard  and  soft  data,  suggesting  a  growing  need  for 
alignment  of  both  (1)  exploited  physical  imagery  products  and  (2)  derived  ontological  textual 
labels  (semantic  markup).  The  authors  first  reviewed  ontology  research  for  situation  awareness, 
information  fusion  and  surveillance,  and  then  focused  on  hard-soft  fusion,  hard-soft  ontology 
fusion  and  geospatial  ontologies. 

They  proposed  an  example  of  hard-soft  ontology  on  Geographical  Information  System  (GIS)  to 
demonstrate  the  need  for  ontology  alignment  to  assist  uses  for  pragmatic  surveillance.  The 
authors  identified  hard  data  as  the  data  from  physical  sensors  (e.g.,  images),  and  soft  data  as 
the  data  from  semantic  markup  (e.g.  textual  labels). 
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The  ontological  fusion  example  proposed  in  this  paper  consists  of  two  information  systems 
applied  in  a  Maritime  scenario,  each  of  which  collect  and  label  their  products  uniquely.  The 
authors  indicated  the  need  to  align  (or  match)  the  ontology  representations  of  the  physical 
objects  to  the  situation  for  coordination  with  the  image  displays  of  the  coastal  areas  (e.g., 
determining  ‘land  areas’  and  ‘waterways’,  traffic  patterns,  types  of  vessels,  and  vessel  activities 
and  identification  by  a  way  of  associated  behaviors  and  class  labels  from  historical  data). 

Fischer,  Y.  and  Bauer,  A.  (2010).  Object-Oriented  Sensor  Data  Fusion  for  Wide  Maritime 
Surveillance.  Proceedings  of  the  2010  International  Waterside  Security  Conference 
(WSS).  IEEE. 

This  article  introduces  an  object-oriented  approach  for  the  fusion  of  observations  produced  by 
heterogeneous  sensors.  The  authors  reviewed  related  work,  in  particular,  Endsley’s  perception- 
comprehension-projection  theory  and  the  Joint  Directors  of  Laboratories  data  fusion  information 
model,  and  proposed  an  object-oriented  architecture  that  was  implemented  and  tested  with  the 
STEP  Scenario  Simulation  Tool. 

The  proposed  architecture,  known  as  Object-Oriented  World  Model  (OOWM),  comprises  an 
Instance  manager,  a  Data  Association,  a  Fusion  Manager  for  position/speed,  and  length,  and  a 
Classification  module.  The  Observation  Interface  connects  with  various  sensor  services  through 
an  observation  bus  and  an  access  Interface  is  provided  for  application-level  event  reports  or 
queries  for  fusion  results.  A  simple  rule-based  approach  is  included  in  the  architecture  for 
situation  awareness  services. 

Khaleghi,  B.,  Khamis,  A.,  and  Karray,  F.  O.  (2013).  Multi-sensor  data  fusion:  a  review  of 
the  state-of-the-art.  Information  Fusion,  14(1):  28-44. 

This  paper  presents  a  comprehensive  review  of  the  data-fusion  state  of  the  art,  exploring  its 
conceptualizations,  benefits  and  challenging  aspects,  as  well  as  multi-sensor  data  fusion 
algorithms.  They  note  that  uncertainty  in  the  products  of  multi-sensor  data  fusion  may  arise  due 
to  any  of  the  following: 

•  imperfect  data 

•  correlated  data 

•  inconsistent  data 

•  disparate  data 

•  Incomplete  data 

For  the  fusion  of  imperfect  data,  the  authors  summarized  the  algorithms  as  follows: 

•  Probabilistic  fusion  (Bayesian  fusion  is  the  core  method) 

•  Evidential  belief  reasoning  (Dempster-Shafer  theory  is  the  popular  method) 
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•  Fusion  and  fuzzy  reasoning 

•  Possibilistic  fusion  using  fuzzy-set  based  possibility  degree  for  data  fusion 

•  Rough  set  based  fusion 

•  Hybrid  fusion  approaches,  e.g.  hybridization  of  fuzzy  theory  with  Dempster-Shafer  theory, 
and  combination  of  fuzzy  set  theory  with  Rough  set  theory 

•  Random  set  theoretic  fusion 

Two  approaches  for  fusion  of  correlated  data  were  reviewed: 

1 .  eliminating  data  correlation 

a.  explicitly  by  removal  of  data 

b.  implicitly  through  reconstruction  of  measurements 

2.  data  fusion  in  presence  of  unknown  correlations 

a.  Covariance  Intersection  is  the  most  common  fusion  method  to  deal  with  correlated  data. 
The  authors  reviewed  methods  for  fusion  of  inconsistent  data  based  on  three  data  types 

1 .  spurious  data 

2.  out  of  sequence  data 

3.  conflicting  data 

The  majority  of  work  on  treating  spurious  data  has  exploited  Bayesian  fusion  frameworks  to 
identify  or  predict  spurious  data,  and  subsequently  eliminate  these  outliers  from  the  fusion 
process.  An  approximate  sub-optimal  solution  to  out  of  sequence  measurement  (OOSM)  called 
“Algorithm  B”  as  well  as  its  famous  optimal  counterpart  “Algorithm  A”  had  been  proposed.  The 
problem  of  out  of  sequence  data  for  disordered  tracks  (OOST)  is  solved  using  equivalent 
measurements  obtained  from  individual  sensor  tracks,  which  are  then  used  in  an  augmented 
state  framework  to  compute  the  joint  density  of  the  current  target  state  and  the  target  state 
corresponding  to  the  delayed  data.  Two  techniques  for  fusion  of  disparate  data,  the  Dempster- 
Shafer  theoretic  framework  for  soft/hard  data  fusion  and  the  human  centered  data  fusion 
paradigm,  are  briefly  reviewed. 
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